Leela is more Chess-domain specific than regular Chess engines

Discussion of anything and everything relating to chess playing software and machines.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
User avatar
Laskos
Posts: 9545
Joined: Wed Jul 26, 2006 8:21 pm
Full name: Kai Laskos

Leela is more Chess-domain specific than regular Chess engines

Post by Laskos » Thu Aug 09, 2018 12:03 pm

The chosen by A0 and Leela network topology (3x3 convolutions) seems to be ill-adapted for gliders in Chess because many moves are not contained in a small area. That means you need many layers in order to be able to see that Rook c1 attacks Queen c8. And even then you can only see it if many filters in the traversed layers are dedicated to transporting piece locations to the 3x3 cell that can integrate them. So, I tried a variant of the game without gliders, building a small opening book from this starting position:



I was curious to see how Lc0 (ID10480) performs against traditional Chess engines in this variant. And Lc0 under-performs by 100 +/- 30 Elo points compared to the performance in regular Chess against same traditional Chess engines. That came a bit as a surprise.

Then, I imagined that it will perform even worse with lots of heavy gliders and only a few non-gliders, so I built a small opening book from this starting position:



Indeed, Lc0 under-performs in this variant against traditional Chess engines by a whopping 300 +/- 50 Elo points compared to regular Chess performance.

Time control used was short (0.3s/move).

All in all, it is true than 3x3 filters do have some difficulty with heavy pieces in Chess, but another conclusion would be that Leela is more Chess-domain specialized than traditional Chess engines, which perform better in variants deviating seriously from Chess.

Milos
Posts: 3387
Joined: Wed Nov 25, 2009 12:47 am

Re: Leela is more Chess-domain specific than regular Chess engines

Post by Milos » Thu Aug 09, 2018 12:52 pm

Laskos wrote:
Thu Aug 09, 2018 12:03 pm
The chosen by A0 and Leela network topology (3x3 convolutions) seems to be ill-adapted for gliders in Chess because many moves are not contained in a small area. That means you need many layers in order to be able to see that Rook c1 attacks Queen c8. And even then you can only see it if many filters in the traversed layers are dedicated to transporting piece locations to the 3x3 cell that can integrate them. So, I tried a variant of the game without gliders, building a small opening book from this starting position:



I was curious to see how Lc0 (ID10480) performs against traditional Chess engines in this variant. And Lc0 under-performs by 100 +/- 30 Elo points compared to the performance in regular Chess against same traditional Chess engines. That came a bit as a surprise.

Then, I imagined that it will perform even worse with lots of heavy gliders and only a few non-gliders, so I built a small opening book from this starting position:



Indeed, Lc0 under-performs in this variant against traditional Chess engines by a whopping 300 +/- 50 Elo points compared to regular Chess performance.

Time control used was short (0.3s/move).

All in all, it is true than 3x3 filters do have some difficulty with heavy pieces in Chess, but another conclusion would be that Leela is more Chess-domain specialized than traditional Chess engines, which perform better in variants deviating seriously from Chess.
With larger kernels (like 8x8) you loose at least 10x in speed. You think that can be compensated with more knowledge? I don't think so.

User avatar
Laskos
Posts: 9545
Joined: Wed Jul 26, 2006 8:21 pm
Full name: Kai Laskos

Re: Leela is more Chess-domain specific than regular Chess engines

Post by Laskos » Thu Aug 09, 2018 1:31 pm

Milos wrote:
Thu Aug 09, 2018 12:52 pm
With larger kernels (like 8x8) you loose at least 10x in speed. You think that can be compensated with more knowledge? I don't think so.
I am not sure if the speed is the problem, as you might need less layers and smaller nets. A single layer with filters examining 8x1 or 1x8 areas, or diagonals, would do in some cases (not that many, but you get it) the job far more efficiently, and a factor of 10 is possible. I guess the problem is that 3x3 convolutions are very well supported by the community to use on say GPUs, and have huge optimizations done on them. It will probably be very hard and inefficient to use different filters, although not theoretically undoable. My feeling is that Go and image recognition are more adapted to those 3x3 filters than Chess.

Anyway, the surprise was not that Leela has problems with heavy pieces, it was to be expected. I was surprised a bit that Leela is more Chess-domain specific than regular Chess engines with hand-crafted evaluation.

Milos
Posts: 3387
Joined: Wed Nov 25, 2009 12:47 am

Re: Leela is more Chess-domain specific than regular Chess engines

Post by Milos » Thu Aug 09, 2018 1:42 pm

Laskos wrote:
Thu Aug 09, 2018 1:31 pm
Milos wrote:
Thu Aug 09, 2018 12:52 pm
With larger kernels (like 8x8) you loose at least 10x in speed. You think that can be compensated with more knowledge? I don't think so.
I am not sure if the speed is the problem, as you might need less layers and smaller nets. A single layer with filters examining 8x1 or 1x8 areas, or diagonals, would do in some cases (not that many, but you get it) the job far more efficiently, and a factor of 10 is possible. I guess the problem is that 3x3 convolutions are very well supported by the community to use on say GPUs, and have huge optimizations done on them. It will probably be very hard and inefficient to use different filters, although not theoretically undoable. My feeling is that Go and image recognition are more adapted to those 3x3 filters than Chess.

Anyway, the surprise was not that Leela has problems with heavy pieces, it was to be expected. I was surprised a bit that Leela is more Chess-domain specific than regular Chess engines with hand-crafted evaluation.
8x1 or 1x8 kernels even though less complex than 3x3 in terms of multiplication in direct implementation don't have fast implementation and question if it is even possible.
Regarding chess-domain specific knowledge, it would really be astonishing that something that contains 60MB's of data (weights) is not better than something that contains a few KB's at best.
The thing with current hand-crafted evaluation that it is optimal for A/B with strong reductions. It would be quite easy to make much better (and slower) hand-crafted evaluation but it would be useless for A/B search.

jp
Posts: 841
Joined: Mon Apr 23, 2018 5:54 am

Re: Leela is more Chess-domain specific than regular Chess engines

Post by jp » Thu Aug 09, 2018 1:45 pm

Laskos wrote:
Thu Aug 09, 2018 12:03 pm
All in all, it is true than 3x3 filters do have some difficulty with heavy pieces in Chess, but another conclusion would be that Leela is more Chess-domain specialized than traditional Chess engines, which perform better in variants deviating seriously from Chess.
Interesting find, Kai, and your examples are not really even variants, just odd positions.

jkiliani
Posts: 143
Joined: Wed Jan 17, 2018 12:26 pm

Re: Leela is more Chess-domain specific than regular Chess engines

Post by jkiliani » Thu Aug 09, 2018 2:09 pm

Laskos wrote:
Thu Aug 09, 2018 12:03 pm
All in all, it is true than 3x3 filters do have some difficulty with heavy pieces in Chess, but another conclusion would be that Leela is more Chess-domain specialized than traditional Chess engines, which perform better in variants deviating seriously from Chess.
I'm not sure the filters are the problem here, my guess is it's the training: Leela plays well in situations she's used to, while for situations she has never seen, her neural net won't know how to handle them very well. I'm actually surprised she lost to traditional engines only by ~100 Elo here. If you want Leela to play such situations well, I'm afraid this will require training a network through reinforcement learning, from scratch.

Branko Radovanovic
Posts: 59
Joined: Sat Sep 13, 2014 2:12 pm

Re: Leela is more Chess-domain specific than regular Chess engines

Post by Branko Radovanovic » Thu Aug 09, 2018 9:46 pm

Laskos wrote:
Thu Aug 09, 2018 12:03 pm
All in all, it is true than 3x3 filters do have some difficulty with heavy pieces in Chess, but another conclusion would be that Leela is more Chess-domain specialized than traditional Chess engines, which perform better in variants deviating seriously from Chess.
An excellent find and - what is more - the way you found it is true science, as usual!

I suspect Leela may also exhibit other flaws inherent to DNNs, such as vulnerability to "one pixel attack". People tend to attribute its quirkiness to insufficient training, but it may well be that its eval - in its current implementation - will inevitably have a significant degree of noise, no matter how well trained it is. Looking at TCEC Div4 games, what is striking to me is not that NN eval is simply "too optimistic", as many have noted - it's that it often rises significantly as the game progresses, without a meaningful improvement of the position.

Also, your conclusion seems to confirm my feeling that Leela would suffer greatly in Chess960. Obviously, its opening-phase advantage over other engines would be nullified, but it's probably much more than that.

Michel
Posts: 2057
Joined: Sun Sep 28, 2008 11:50 pm

Re: Leela is more Chess-domain specific than regular Chess engines

Post by Michel » Fri Aug 10, 2018 3:49 am

It's a bit disappointing but not unexpected.

http://nautil.us/blog/this-neural-net-h ... ates-sheep

If a NN is confronted with a type of data different from what it is trained on, it will fail spectacularly.
Ideas=science. Simplification=engineering.
Without ideas there is nothing to simplify.

Gian-Carlo Pascutto
Posts: 1191
Joined: Sat Dec 13, 2008 6:00 pm
Contact:

Re: Leela is more Chess-domain specific than regular Chess engines

Post by Gian-Carlo Pascutto » Fri Aug 10, 2018 8:33 am

Branko Radovanovic wrote:
Thu Aug 09, 2018 9:46 pm
Looking at TCEC Div4 games, what is striking to me is not that NN eval is simply "too optimistic", as many have noted - it's that it often rises significantly as the game progresses, without a meaningful improvement of the position.
This has more to do with the temperature used in training than anything, I believe. It is trained to expect opponent blunders.

Having watched many games, I am not sure if it is bad or not. She wins many games she shouldn't have, because she's better at judging when the opponent's position is harder to play.

Gian-Carlo Pascutto
Posts: 1191
Joined: Sat Dec 13, 2008 6:00 pm
Contact:

Re: Leela is more Chess-domain specific than regular Chess engines

Post by Gian-Carlo Pascutto » Fri Aug 10, 2018 8:34 am

Branko Radovanovic wrote:
Thu Aug 09, 2018 9:46 pm
Also, your conclusion seems to confirm my feeling that Leela would suffer greatly in Chess960. Obviously, its opening-phase advantage over other engines would be nullified, but it's probably much more than that.
If you train it to Chess960 too, it'd work fine.

If I write a classical engine and make it assume castling goes to c1 and g1, it's going to have some problems in Chess960 too.

Post Reply