Leela is more Chess-domain specific than regular Chess engines

[corres]

Maybe the machine learning advances came exactly because processing power increased dramatically, as the main ideas (like the back-propagation) are from 1980s. The advances with machine learning are still impressive, as in many specialized, well-defined tasks, the approach achieves superlative results.

So we use AB engines for a long time too...

[Laskos]

The RL approach builds a giant evaluation function based on pattern recognition. The positions you present here are obviously irregular patterns and LC0 can make no real use of the pattern recognition.

On the other hand, regular engines "break-down" the evaluation function into logical pieces of information and can easily adapt to more strange positions.

That is expected.

If you train LC0 with more irregular patterns it would excel there too.

Well put. In the example with 4 knights, LC0 has basically no experience in positions with a 4th knight. Unlike Stockfish, it may not be giving any score advantage to having a 4th knight, since it is never been trained on that type of position and may not realize that 4 knights is probably better than 3, while Stockfish happily adds 300cp for every knight you stick on the board.

Well, it seems I was a bit uninformedly optimistic about NNs flexibility and abstraction power, to be disappointed by this very specialized solver of very specific, well-defined tasks. For those 4 presented positions, A0 or Lc0 would probably need to be heavily trained on humongous amounts of specialized data for each of them, to be competitive with one Chess-specific regular engine. So, "zero" approach, which might seem ab-initio generalistic problem solver, seems more of a hype to me now as approaching a generalized AI. And in fact, maybe "Zillions of Games", a general game playing system developed by Mark Lefler and Jeff Mallett in 1998, which uses just some "rules files" (computationally almost 0 input), might impress me more as generalized AI than these big data, big hardware expert systems with fancy DCNNs black-boxes approaches. Maybe the machine learning advances came exactly because processing power increased dramatically, as the main ideas (like the back-propagation) are from 1980s. The advances with machine learning are still impressive, as in many specialized, well-defined tasks, the approach achieves superlative results.

As this 30xxx run seems to like very much the French Defense (probably arbitrarily so), I checked that by training more on God-sent to Leela French Defense, it performs better there. It does:

From general PGN 6-ply openings:
General
Score of lc0_v19_31608 vs SF_10: 20 - 92 - 88 [0.320] 200
Elo difference: -130.94 +/- 36.43
Finished match


From Main Line French, with a small PGN 4-ply book:
French: Main Line
Score of lc0_v19_31608 vs SF_10: 38 - 52 - 110 [0.465] 200
Elo difference: -24.36 +/- 32.33
Finished match

It overperforms in French by 100+ Elo points.

I totally agree with my post I replied to .
Heck, these same techniques Leela uses, and even more advanced ones, had hard time with Pac-Man, IIRC.

[Javier Ros]

The RL approach builds a giant evaluation function based on pattern recognition. The positions you present here are obviously irregular patterns and LC0 can make no real use of the pattern recognition.

On the other hand, regular engines "break-down" the evaluation function into logical pieces of information and can easily adapt to more strange positions.

That is expected.

If you train LC0 with more irregular patterns it would excel there too.

Well put. In the example with 4 knights, LC0 has basically no experience in positions with a 4th knight. Unlike Stockfish, it may not be giving any score advantage to having a 4th knight, since it is never been trained on that type of position and may not realize that 4 knights is probably better than 3, while Stockfish happily adds 300cp for every knight you stick on the board.

Well, it seems I was a bit uninformedly optimistic about NNs flexibility and abstraction power, to be disappointed by this very specialized solver of very specific, well-defined tasks. For those 4 presented positions, A0 or Lc0 would probably need to be heavily trained on humongous amounts of specialized data for each of them, to be competitive with one Chess-specific regular engine. So, "zero" approach, which might seem ab-initio generalistic problem solver, seems more of a hype to me now as approaching a generalized AI. And in fact, maybe "Zillions of Games", a general game playing system developed by Mark Lefler and Jeff Mallett in 1998, which uses just some "rules files" (computationally almost 0 input), might impress me more as generalized AI than these big data, big hardware expert systems with fancy DCNNs black-boxes approaches. Maybe the machine learning advances came exactly because processing power increased dramatically, as the main ideas (like the back-propagation) are from 1980s. The advances with machine learning are still impressive, as in many specialized, well-defined tasks, the approach achieves superlative results.

As this 30xxx run seems to like very much the French Defense (probably arbitrarily so), I checked that by training more on God-sent to Leela French Defense, it performs better there. It does:

From general PGN 6-ply openings:
General
Score of lc0_v19_31608 vs SF_10: 20 - 92 - 88 [0.320] 200
Elo difference: -130.94 +/- 36.43
Finished match


From Main Line French, with a small PGN 4-ply book:
French: Main Line
Score of lc0_v19_31608 vs SF_10: 38 - 52 - 110 [0.465] 200
Elo difference: -24.36 +/- 32.33
Finished match

It overperforms in French by 100+ Elo points.

I totally agree with my post I replied to .
Heck, these same techniques Leela uses, and even more advanced ones, had hard time with Pac-Man, IIRC.

I have detected in my tests this superiority of lc0 in the French Defense. This superiority can be due to several factors:

1) lc0 has trained a big proportion of its train games with this opening.
2) Stockfish, as the rest of alpha-beta programs, doesn't understand closed positions like the derived from the French.
3) The positional style of lc0 is the best to play this opening.

Which factor do you think is the most important?

May you give the information about scores of Black and White pieces for lc0 and SF?

I have played French Defense during some years and my conclusion is that it is inferior to other lines like Sicilian Defense, that's why I'm struck by the fact that lc0 trained so much that opening.

[Guenther]

I have detected in my tests this superiority of lc0 in the French Defense. This superiority can be due to several factors:

1) lc0 has trained a big proportion of its train games with this opening.
2) Stockfish, as the rest of alpha-beta programs, doesn't understand closed positions like the derived from the French.
3) The positional style of lc0 is the best to play this opening.

Which factor do you think is the most important?

May you give the information about scores of Black and White pieces for lc0 and SF?

I have played French Defense during some years and my conclusion is that it is inferior to other lines like Sicilian Defense, that's why I'm struck by the fact that lc0 trained so much that opening.

It is 1) for NN30

[chrisw]

1/ Leela underperforms in odd positions

This is expected, with a style rarely or never encountered in training are harder for Leela than those it has experience with.

Why this is expected? I expected that the NN topology (3x3 kernels) used in image recognition will have problems with gliders in Chess.

Yes, that is so. Deepmind (and LC0) endeavour to get round this “problem” by hurling many many layers at it. But this is computationally expensive (Deepmind didn’t care, I guess, they had the hardware available, so ...), there are a couple (that I read of) of smarter ways to approach the problem, so all is not lost for those of us with only a couple of GPU’s available.

But that DCNN based engine is more specialized in Chess than engines using hand-crafted specific Chess knowledge is not necessarily expected. The net is not a book, and using those weights of the black-box gives SOME level of abstraction and generalization, but this level seems low, which is not necessarily expected. In fact having lower power of inference and generalization than traditional engines is counter-intuitive to me, that is why I was disappointed. Although the "zero approach" seems generalistic, its final results are some sort of expert systems, which hit their efficiency punctually in some well defined very specialized problems.

2/ It under-performs significantly more with lots of gliders

Perhaps, but is this really something you can conclude based on your test?

I don't understand. That Leela performs significantly worse against traditional engines in positions with many heavy gliders is a fact, or maybe, like Leela, you need a million examples to learn this?

It seems, the 3x3 patterns in consecutive layers used are not adapted extremely well to Chess.

This is where you are making a leap, imo. With 4 CNN layers, a feature in Leela's net can take into account information from the entire board. Leela has 10 times that many CNN layers.

There are tons of reasons Leela could be under-performing in those variants. Perhaps AB engines are just exceptionally strong in those variants? Perhaps large portions of Leela's nets deal with interactions between major and minor pieces? Maybe search is very important in those variants and Leela is suffering from lack of NPS? Why do you conclude that it is the fact that Leela uses 3x3 filters that she is under-performing there?

I think the weakness with discovered threats and pins is well known with Leela. They involve gliders again, and as shown here, with lots of gliders Leela generally seems to underperform badly against traditional engines. It indicated that 3x3 filters might not be very efficient with 8x1 rays. And if traditional Chess engines, with hand-crafted Chess-specific eval, are "exceptionally strong in those variants", then it's a pretty bad omen for the used approach with DNNs as a way to a generalized AI. Also, the weakness with gliders and the "single pixel attack" might be the main reasons for very weak tactical abilities of Leela.