Page 1 of 4
Leela is more Chess-domain specific than regular Chess engines
Posted: Thu Aug 09, 2018 2:03 pm
by Laskos
The chosen by A0 and Leela network topology (3x3 convolutions) seems to be ill-adapted for gliders in Chess because many moves are not contained in a small area. That means you need many layers in order to be able to see that Rook c1 attacks Queen c8. And even then you can only see it if many filters in the traversed layers are dedicated to transporting piece locations to the 3x3 cell that can integrate them. So, I tried a variant of the game without gliders, building a small opening book from this starting position:
[d]2nknnn1/2pppppp/8/8/8/8/PPPPPP2/1NNNKN2 w - - 0 1
I was curious to see how Lc0 (ID10480) performs against traditional Chess engines in this variant. And Lc0 under-performs by 100 +/- 30 Elo points compared to the performance in regular Chess against same traditional Chess engines. That came a bit as a surprise.
Then, I imagined that it will perform even worse with lots of heavy gliders and only a few non-gliders, so I built a small opening book from this starting position:
[d]2qkq3/2rrr3/1pppp3/8/8/3PPPP1/3RRR2/3QKQ2 w - - 0 1
Indeed, Lc0 under-performs in this variant against traditional Chess engines by a whopping 300 +/- 50 Elo points compared to regular Chess performance.
Time control used was short (0.3s/move).
All in all, it is true than 3x3 filters do have some difficulty with heavy pieces in Chess, but another conclusion would be that Leela is more Chess-domain specialized than traditional Chess engines, which perform better in variants deviating seriously from Chess.
Re: Leela is more Chess-domain specific than regular Chess engines
Posted: Thu Aug 09, 2018 2:52 pm
by Milos
Laskos wrote: ↑Thu Aug 09, 2018 2:03 pm
The chosen by A0 and Leela network topology (3x3 convolutions) seems to be ill-adapted for gliders in Chess because many moves are not contained in a small area. That means you need many layers in order to be able to see that Rook c1 attacks Queen c8. And even then you can only see it if many filters in the traversed layers are dedicated to transporting piece locations to the 3x3 cell that can integrate them. So, I tried a variant of the game without gliders, building a small opening book from this starting position:
[d]2nknnn1/2pppppp/8/8/8/8/PPPPPP2/1NNNKN2 w - - 0 1
I was curious to see how Lc0 (ID10480) performs against traditional Chess engines in this variant. And
Lc0 under-performs by 100 +/- 30 Elo points compared to the performance in regular Chess against same traditional Chess engines. That came a bit as a surprise.
Then, I imagined that it will perform even worse with lots of heavy gliders and only a few non-gliders, so I built a small opening book from this starting position:
[d]2qkq3/2rrr3/1pppp3/8/8/3PPPP1/3RRR2/3QKQ2 w - - 0 1
Indeed,
Lc0 under-performs in this variant against traditional Chess engines by a whopping 300 +/- 50 Elo points compared to regular Chess performance.
Time control used was short (0.3s/move).
All in all, it is true than 3x3 filters do have some difficulty with heavy pieces in Chess, but another conclusion would be that Leela is more Chess-domain specialized than traditional Chess engines, which perform better in variants deviating seriously from Chess.
With larger kernels (like 8x8) you loose at least 10x in speed. You think that can be compensated with more knowledge? I don't think so.
Re: Leela is more Chess-domain specific than regular Chess engines
Posted: Thu Aug 09, 2018 3:31 pm
by Laskos
Milos wrote: ↑Thu Aug 09, 2018 2:52 pm
With larger kernels (like 8x8) you loose at least 10x in speed. You think that can be compensated with more knowledge? I don't think so.
I am not sure if the speed is the problem, as you might need less layers and smaller nets. A single layer with filters examining 8x1 or 1x8 areas, or diagonals, would do in some cases (not that many, but you get it) the job far more efficiently, and a factor of 10 is possible. I guess the problem is that 3x3 convolutions are very well supported by the community to use on say GPUs, and have huge optimizations done on them. It will probably be very hard and inefficient to use different filters, although not theoretically undoable. My feeling is that Go and image recognition are more adapted to those 3x3 filters than Chess.
Anyway, the surprise was not that Leela has problems with heavy pieces, it was to be expected. I was surprised a bit that Leela is more Chess-domain specific than regular Chess engines with hand-crafted evaluation.
Re: Leela is more Chess-domain specific than regular Chess engines
Posted: Thu Aug 09, 2018 3:42 pm
by Milos
Laskos wrote: ↑Thu Aug 09, 2018 3:31 pm
Milos wrote: ↑Thu Aug 09, 2018 2:52 pm
With larger kernels (like 8x8) you loose at least 10x in speed. You think that can be compensated with more knowledge? I don't think so.
I am not sure if the speed is the problem, as you might need less layers and smaller nets. A single layer with filters examining 8x1 or 1x8 areas, or diagonals, would do in some cases (not that many, but you get it) the job far more efficiently, and a factor of 10 is possible. I guess the problem is that 3x3 convolutions are very well supported by the community to use on say GPUs, and have huge optimizations done on them. It will probably be very hard and inefficient to use different filters, although not theoretically undoable. My feeling is that Go and image recognition are more adapted to those 3x3 filters than Chess.
Anyway, the surprise was not that Leela has problems with heavy pieces, it was to be expected. I was surprised a bit that Leela is more Chess-domain specific than regular Chess engines with hand-crafted evaluation.
8x1 or 1x8 kernels even though less complex than 3x3 in terms of multiplication in direct implementation don't have fast implementation and question if it is even possible.
Regarding chess-domain specific knowledge, it would really be astonishing that something that contains 60MB's of data (weights) is not better than something that contains a few KB's at best.
The thing with current hand-crafted evaluation that it is optimal for A/B with strong reductions. It would be quite easy to make much better (and slower) hand-crafted evaluation but it would be useless for A/B search.
Re: Leela is more Chess-domain specific than regular Chess engines
Posted: Thu Aug 09, 2018 3:45 pm
by jp
Laskos wrote: ↑Thu Aug 09, 2018 2:03 pm
All in all, it is true than 3x3 filters do have some difficulty with heavy pieces in Chess, but another conclusion would be that Leela is more Chess-domain specialized than traditional Chess engines, which perform better in variants deviating seriously from Chess.
Interesting find, Kai, and your examples are not really even variants, just odd positions.
Re: Leela is more Chess-domain specific than regular Chess engines
Posted: Thu Aug 09, 2018 4:09 pm
by jkiliani
Laskos wrote: ↑Thu Aug 09, 2018 2:03 pm
All in all, it is true than 3x3 filters do have some difficulty with heavy pieces in Chess, but another conclusion would be that Leela is more Chess-domain specialized than traditional Chess engines, which perform better in variants deviating seriously from Chess.
I'm not sure the filters are the problem here, my guess is it's the training: Leela plays well in situations she's used to, while for situations she has never seen, her neural net won't know how to handle them very well. I'm actually surprised she lost to traditional engines only by ~100 Elo here. If you want Leela to play such situations well, I'm afraid this will require training a network through reinforcement learning, from scratch.
Re: Leela is more Chess-domain specific than regular Chess engines
Posted: Thu Aug 09, 2018 11:46 pm
by Branko Radovanovic
Laskos wrote: ↑Thu Aug 09, 2018 2:03 pm
All in all, it is true than 3x3 filters do have some difficulty with heavy pieces in Chess, but another conclusion would be that Leela is more Chess-domain specialized than traditional Chess engines, which perform better in variants deviating seriously from Chess.
An excellent find and - what is more - the way you found it is true science, as usual!
I suspect Leela may also exhibit other flaws inherent to DNNs, such as vulnerability to "one pixel attack". People tend to attribute its quirkiness to insufficient training, but it may well be that its eval - in its current implementation - will inevitably have a significant degree of noise, no matter how well trained it is. Looking at TCEC Div4 games, what is striking to me is not that NN eval is simply "too optimistic", as many have noted - it's that it often rises significantly as the game progresses, without a meaningful improvement of the position.
Also, your conclusion seems to confirm my feeling that Leela would suffer greatly in Chess960. Obviously, its opening-phase advantage over other engines would be nullified, but it's probably much more than that.
Re: Leela is more Chess-domain specific than regular Chess engines
Posted: Fri Aug 10, 2018 5:49 am
by Michel
It's a bit disappointing but not unexpected.
http://nautil.us/blog/this-neural-net-h ... ates-sheep
If a NN is confronted with a type of data different from what it is trained on, it will fail spectacularly.
Re: Leela is more Chess-domain specific than regular Chess engines
Posted: Fri Aug 10, 2018 10:33 am
by Gian-Carlo Pascutto
Branko Radovanovic wrote: ↑Thu Aug 09, 2018 11:46 pm
Looking at TCEC Div4 games, what is striking to me is not that NN eval is simply "too optimistic", as many have noted - it's that it often rises significantly as the game progresses, without a meaningful improvement of the position.
This has more to do with the temperature used in training than anything, I believe. It is trained to expect opponent blunders.
Having watched many games, I am not sure if it is bad or not. She wins many games she shouldn't have, because she's better at judging when the opponent's position is harder to play.
Re: Leela is more Chess-domain specific than regular Chess engines
Posted: Fri Aug 10, 2018 10:34 am
by Gian-Carlo Pascutto
Branko Radovanovic wrote: ↑Thu Aug 09, 2018 11:46 pm
Also, your conclusion seems to confirm my feeling that Leela would suffer greatly in Chess960. Obviously, its opening-phase advantage over other engines would be nullified, but it's probably much more than that.
If you train it to Chess960 too, it'd work fine.
If I write a classical engine and make it assume castling goes to c1 and g1, it's going to have some problems in Chess960 too.