jorose wrote: ↑Sat Aug 11, 2018 8:37 am
Laskos wrote: ↑Sat Aug 11, 2018 2:23 am
1/ Leela underperforms in odd positions
This is expected, with a style rarely or never encountered in training are harder for Leela than those it has experience with.
Why this is expected? I expected that the NN topology (3x3 kernels) used in image recognition will have problems with gliders in Chess. But that DCNN based engine is more specialized in Chess than engines using hand-crafted specific Chess knowledge is not necessarily expected. The net is not a book, and using those weights of the black-box gives SOME level of abstraction and generalization, but this level seems low, which is not necessarily expected. In fact having lower power of inference and generalization than traditional engines is counter-intuitive to me, that is why I was disappointed. Although the "zero approach" seems generalistic, its final results are some sort of expert systems, which hit their efficiency punctually in some well defined very specialized problems.
Laskos wrote: ↑Sat Aug 11, 2018 2:23 am
2/ It under-performs significantly more with lots of gliders
Perhaps, but is this really something you can conclude based on your test?
I don't understand. That Leela performs significantly worse against traditional engines in positions with many heavy gliders is a fact, or maybe, like Leela, you need a million examples to learn this?
Laskos wrote: ↑Sat Aug 11, 2018 2:23 am
It seems, the 3x3 patterns in consecutive layers used are not adapted extremely well to Chess.
This is where you are making a leap, imo. With 4 CNN layers, a feature in Leela's net can take into account information from the entire board. Leela has 10 times that many CNN layers.
There are tons of reasons Leela could be under-performing in those variants. Perhaps AB engines are just exceptionally strong in those variants? Perhaps large portions of Leela's nets deal with interactions between major and minor pieces? Maybe search is very important in those variants and Leela is suffering from lack of NPS? Why do you conclude that it is the fact that Leela uses 3x3 filters that she is under-performing there?
I think the weakness with discovered threats and pins is well known with Leela. They involve gliders again, and as shown here, with lots of gliders Leela generally seems to underperform badly against traditional engines. It indicated that 3x3 filters might not be very efficient with 8x1 rays. And if traditional Chess engines, with hand-crafted Chess-specific eval, are "exceptionally strong in those variants", then it's a pretty bad omen for the used approach with DNNs as a way to a generalized AI. Also, the weakness with gliders and the "single pixel attack" might be the main reasons for very weak tactical abilities of Leela.
