Is AlphaZero-LC0 (Leela0) A chess Engine?

[supersharp77]

Now before anyone jumps to any conclusions regarding the 'headline' of this post I would like to make a statement: I love what the LC0 (Leela0) github project has accomplished ..love the binaries..the whole concept is fascinating and very exciting and all that BUT I must keep in mind that I have been testing Chess Programs and Engines since the Chess Challenger..Mephisto Saitek..Novag Constellation hardware and have heard all kinds of claims over the years..Lets go back to the beginning Dec 2017 When Google Deep Mind claimed that AlphaZero 'learned chess and beat Stockfish in under 24 hrs' .."Then Dropped out of sight and under the radar" That's the Supercharged environment we inherited..meanwhile we the engine testing community were left with the task of defending our claims that Stockfish (or Houdini or Komodo) where the "strongest chess engines in the world"...Since that time back in December AlphaZero has never entered a chess engine competition (to my knowledge) although recently I believe they did release some additional games vs Stockfish (still without opening books I believe)...Ok thats some historical context..Cut to the current day...Downloaded the LC0 (Leela0) chess engines read some blogs..Wow..engine starts playing games...engine wins some games..engine plays moves in Lokasoft, Shredder and Chessbase Gui's with no problems..but engine does not look capable of beating Stockfish in any of the gui's unless a certain type of position is reached over the board..engine seems to excel in complex maneuvering types of positions but misses obvious tactical trick and motifs..what to think? what to do? Engine builds up "fantastic positions" then blunders like a child..what to think what to do? It got me to thinking..If the AlphaZero-Leela0(LC0) concept depends on "trial & error ("playing hundreds of thousands or millions or billions of games to learn all the right and wrong moves to play ") It hit me that "even an idiot with a photographic memory could become a SuperGrandmaster if the idiot played enough chess games to remember "all the right moves from the wrong moves" But as far as I know that has never happened...Thats why LC0(Leela0) AlphaZero needs hyper-speed to peruse large volumes of games to "find" the right moves to play...The problem in chess is the "Dreaded Chess Novelty" That new move that is not in any chess book..the in between move...Piece rearrangement during the game..the unusual midgame or ending...soon the Neural database is empty...no more moves..."a blank canvas" What to do? Does it make AlphaZero/LC0 a chess engine or a chess program or something else? Let me know your thoughts on the matter. Thx AR

[Dann Corbit]

It does not work like that. It starts with the rules of the game.
It does self play to learn what things are good and what things are bad.
It is not memorizing board positions.
It writes out a file of numbers called a network that contains the values for different board features.
It uses this network to judge board positions.

The MCTS sampling is just a search method where it looks at future positions (alpha-beta also does this, but not by sampling).

It is a different approach. But it is still a chess engine.

[gaard]

If you have been involved in computer chess for that long you will likely remember the same criticism being directed towards conventional chess engines. 15 years ago, maybe 20, many people dismissed them as stupid "bean counters". I also remember hearing then and today that conventional chess engines play "ugly chess". Tactically they were already incredibly strong two decades ago but long-term strategizing and positional evaluation was seemingly absent. These newer paradigms are crude as well but it doesn't look like they're having much trouble catching up.

[Albert Silver]

Now before anyone jumps to any conclusions regarding the 'headline' of this post I would like to make a statement: I love what the LC0 (Leela0) github project has accomplished ..love the binaries..the whole concept is fascinating and very exciting and all that BUT I must keep in mind that I have been testing Chess Programs and Engines since the Chess Challenger..Mephisto Saitek..Novag Constellation hardware and have heard all kinds of claims over the years..Lets go back to the beginning Dec 2017 When Google Deep Mind claimed that AlphaZero 'learned chess and beat Stockfish in under 24 hrs' .."Then Dropped out of sight and under the radar" That's the Supercharged environment we inherited..meanwhile we the engine testing community were left with the task of defending our claims that Stockfish (or Houdini or Komodo) where the "strongest chess engines in the world"...Since that time back in December AlphaZero has never entered a chess engine competition (to my knowledge) although recently I believe they did release some additional games vs Stockfish (still without opening books I believe)...Ok thats some historical context..Cut to the current day...Downloaded the LC0 (Leela0) chess engines read some blogs..Wow..engine starts playing games...engine wins some games..engine plays moves in Lokasoft, Shredder and Chessbase Gui's with no problems..but engine does not look capable of beating Stockfish in any of the gui's unless a certain type of position is reached over the board..engine seems to excel in complex maneuvering types of positions but misses obvious tactical trick and motifs..what to think? what to do? Engine builds up "fantastic positions" then blunders like a child..what to think what to do? It got me to thinking..If the AlphaZero-Leela0(LC0) concept depends on "trial & error ("playing hundreds of thousands or millions or billions of games to learn all the right and wrong moves to play ") It hit me that "even an idiot with a photographic memory could become a SuperGrandmaster if the idiot played enough chess games to remember "all the right moves from the wrong moves" But as far as I know that has never happened...Thats why LC0(Leela0) AlphaZero needs hyper-speed to peruse large volumes of games to "find" the right moves to play...The problem in chess is the "Dreaded Chess Novelty" That new move that is not in any chess book..the in between move...Piece rearrangement during the game..the unusual midgame or ending...soon the Neural database is empty...no more moves..."a blank canvas" What to do? Does it make AlphaZero/LC0 a chess engine or a chess program or something else? Let me know your thoughts on the matter. Thx AR

I guess the first question is what GPU you are using, and what the NPS (nodes per second) ratio was with SF in your tests. It is not at all clear from this post whether you understand that to leverage the strength of Leela you need a GPU of decent standard, and that Deep Mind's claims also hinge on the conditions, not least of which is that they ran AlphaZero at roughly a 900 to 1 NPS handicap.

[sincerelybrent]

...Deep Mind's claims also hinge on the conditions, not least of which is that they ran AlphaZero at roughly a 900 to 1 NPS handicap.

Not a fair comparison of the engines at all. I wonder how it would have gone if the hardware were more comparable.

When I use dkappe’s Leela ratio to equalize GPU to CPU, Lc0 20.2 nn32930 scores dead even vs Stockfish 9 after 100 games of the Silver Suite, each engine playing both sides. Score was +9/=72/-9. When I let it run without equalizing, Lc0 blows Stockfish away. Same conditions: +35/=56/-9. Time controls were 40/2’ repeating.

[mclane]

The question is wrong. It must be: do they play chess.

[Jouni]

I don't like Leela ratio. My suggestions: 1. CPU and GPU same price 2. Best possible CPU/GPU you can get. 3. Max 1000 € readybuilt gaming PC. Fair or not !?

[Albert Silver]

I don't like Leela ratio. My suggestions: 1. CPU and GPU same price 2. Best possible CPU/GPU you can get. 3. Max 1000 € readybuilt gaming PC. Fair or not !?

Not sure what this changes, except the strange 3) you placed. Also, prices will vary quite a bit from continent to continent. So I looked up the price in Europe of the AMD 16-core Threadripper which retails for €879 on Amazon.de, and the equivalent RTX2080 goes for about €800, which should allow the top Leela to break even with SF10 on that CPU. Mind you, if money is used best, then two RTX2060s, also for about €800 (total) will likely beat SF10 on that same 16-core processor.

[Albert Silver]

...Deep Mind's claims also hinge on the conditions, not least of which is that they ran AlphaZero at roughly a 900 to 1 NPS handicap.

Not a fair comparison of the engines at all. I wonder how it would have gone if the hardware were more comparable.

When I use dkappe’s Leela ratio to equalize GPU to CPU, Lc0 20.2 nn32930 scores dead even vs Stockfish 9 after 100 games of the Silver Suite, each engine playing both sides. Score was +9/=72/-9. When I let it run without equalizing, Lc0 blows Stockfish away. Same conditions: +35/=56/-9. Time controls were 40/2’ repeating.

I don't really know what fair means here. I am merely trying to establish a common ground baseline. Since AlphaZero was brought up, and by extension its results, you need to use the conditions Deep Mind used. Otherwise you will end up comparing apples to oranges. We know the reported results of AlphaZero, and we know what the conditions were. In your setup, you are saying that your GPU outperforms the CPU relative to that ratio, which is fine. It wont be the case for everyone though, which is why it is essential to establish a baseline that can be referred to. I.e. the NN is running in conditions better than Deep Mind offered AlphaZero, or worse.

[sincerelybrent]

... it is essential to establish a baseline that can be referred to. I.e. the NN is running in conditions better than Deep Mind offered AlphaZero, or worse.

I meant it wasn't fair in regards to the hardware imbalance and the amount of "time" each engine had to think.

I completely agree a common ground baseline needs to be established, or the results are meaningless. Apples-to-oranges is a good analogy. Without a baseline, there are two variables in the equation, instead of one: a difference in hardware (kNs) and the engines. That's what my point about Lc0 vs Stockfish was meant to highlight. When I used DKappe's LeelaRatio tool and adjusted my settings to get the ratio to 1.0 (1:1), the 100-game score was tied between Lc0 and Stockfish. When I ran the same engines with the same books and settings, w/o adjusting my ratio (which was 3.1, in favor of Leela), Leela "crushed" Stockfish by winning many more games.