Official: Lc0 is the strongest engine :)

[Laskos]

On my fair price-wise hardware in openings and middlegames.

My GPU is GTX 1060 6GB and CPU is 4 core i7 4790 at 3.8 GHz.
Lc0 18.1 ID11261 is using 2 CPU threads and GPU
Stockfish_dev is using 4 CPU cores (4 threads).

Seeing how badly Lc0 underperforms in endgames (more than 350 Elo points weaker than the general play)
http://talkchess.com/forum3/viewtopic.p ... 9&start=33 ,
I used special adjudication rules to avoid endgames, and to see which engine achieves more often a decisive advantage from early openings all through to late middlegames.

• Adjudication:
  
  As Win if both engines show 150cp or more advantage for one engine for 2 consecutive moves. It corresponds to SF_dev Win rate of 90% in openings and middlegames (endgames need a bit higher cp value for 90% Win rate).
  
  As sudden Draw at move 35, so most endgames don't enter the game.

SF_dev uses Contempt=0, as with non-zero contempt it shows fake value of eval. Lc0 has no contempt setting.

Result in 40 games from 3moves_GM_04.epd EPD opening 3-mover suite, which disfavors Lc0, because it's not a PGN opening suite to fill Lc0 history planes. Positions are random, balanced, from GM games. Time control is 60''+1''.

Code: Select all

Games Completed = 40 of 40 (Avg game length = 144.713 sec)
Settings = Gauntlet/128MB/60000ms+1000ms/M 150cp for 2 moves, D 35 moves/EPD:C:\LittleBlitzer\3moves_GM_04.epd(817)
Time = 5900 sec elapsed, 0 sec remaining
 1.  SF_dev 4 cores            	18.5/40	5-8-27  	(L: m=0 t=0 i=0 a=8)	(D: r=0 i=0 f=0 s=0 a=27)	(tpm=2341.3 d=23.37 nps=6118319)
 2.  lc0_v18.1_ID11261        	21.5/40	8-5-27  	(L: m=0 t=0 i=0 a=5)	(D: r=0 i=0 f=0 s=0 a=27)	(tpm=2152.9 d=8.28 nps=2958)
 

Lc0 wins +8 -5 =27.

The next match is against Komodo 12.1.1.

[Laskos]

On my fair price-wise hardware in openings and middlegames.

My GPU is GTX 1060 6GB and CPU is 4 core i7 4790 at 3.8 GHz.
Lc0 18.1 ID11261 is using 2 CPU threads and GPU
Stockfish_dev is using 4 CPU cores (4 threads).

Seeing how badly Lc0 underperforms in endgames (more than 350 Elo points weaker than the general play)
http://talkchess.com/forum3/viewtopic.p ... 9&start=33 ,
I used special adjudication rules to avoid endgames, and to see which engine achieves more often a decisive advantage from early openings all through to late middlegames.

• Adjudication:
  
  As Win if both engines show 150cp or more advantage for one engine for 2 consecutive moves. It corresponds to SF_dev Win rate of 90% in openings and middlegames (endgames need a bit higher cp value for 90% Win rate).
  
  As sudden Draw at move 35, so most endgames don't enter the game.

SF_dev uses Contempt=0, as with non-zero contempt it shows fake value of eval. Lc0 has no contempt setting.

Result in 40 games from 3moves_GM_04.epd EPD opening 3-mover suite, which disfavors Lc0, because it's not a PGN opening suite to fill Lc0 history planes. Positions are random, balanced, from GM games. Time control is 60''+1''.

Code: Select all

Games Completed = 40 of 40 (Avg game length = 144.713 sec)
Settings = Gauntlet/128MB/60000ms+1000ms/M 150cp for 2 moves, D 35 moves/EPD:C:\LittleBlitzer\3moves_GM_04.epd(817)
Time = 5900 sec elapsed, 0 sec remaining
 1.  SF_dev 4 cores            	18.5/40	5-8-27  	(L: m=0 t=0 i=0 a=8)	(D: r=0 i=0 f=0 s=0 a=27)	(tpm=2341.3 d=23.37 nps=6118319)
 2.  lc0_v18.1_ID11261        	21.5/40	8-5-27  	(L: m=0 t=0 i=0 a=5)	(D: r=0 i=0 f=0 s=0 a=27)	(tpm=2152.9 d=8.28 nps=2958)
 

Lc0 wins +8 -5 =27.

The next match is against Komodo 12.1.1.

Similar result against Komodo 12.1.1, even a bit better for Lc0. The sole change is adjudication at 120cp, as 120cp-130cp is the 90% threshold Win rate for Komodo 12.1.1 in these conditions. Contempt=0 is used again for Komodo.

Code: Select all

Games Completed = 40 of 40 (Avg game length = 150.639 sec)
Settings = Gauntlet/128MB/60000ms+1000ms/M 120cp for 2 moves, D 35 moves/EPD:C:\LittleBlitzer\3moves_GM_04.epd(817)
Time = 6140 sec elapsed, 0 sec remaining
 1.  Komodo 12.1.1 4 cores     	18.0/40	4-8-28  	(L: m=0 t=0 i=0 a=8)	(D: r=5 i=0 f=0 s=0 a=23)	(tpm=2582.3 d=22.13 nps=5798291)
 2.  lc0_v18.1_ID11261        	22.0/40	8-4-28  	(L: m=0 t=0 i=0 a=4)	(D: r=5 i=0 f=0 s=0 a=23)	(tpm=2138.9 d=8.37 nps=3148)
 

Lc0 wins +8 -4 =28.

[corres]

On my fair price-wise hardware in openings and middlegames.

My GPU is GTX 1060 6GB and CPU is 4 core i7 4790 at 3.8 GHz.
Lc0 18.1 ID11261 is using 2 CPU threads and GPU
Stockfish_dev is using 4 CPU cores (4 threads).

Seeing how badly Lc0 underperforms in endgames (more than 350 Elo points weaker than the general play)
http://talkchess.com/forum3/viewtopic.p ... 9&start=33 ,
I used special adjudication rules to avoid endgames, and to see which engine achieves more often a decisive advantage from early openings all through to late middlegames.

• Adjudication:
  
  As Win if both engines show 150cp or more advantage for one engine for 2 consecutive moves. It corresponds to SF_dev Win rate of 90% in openings and middlegames (endgames need a bit higher cp value for 90% Win rate).
  
  As sudden Draw at move 35, so most endgames don't enter the game.

SF_dev uses Contempt=0, as with non-zero contempt it shows fake value of eval. Lc0 has no contempt setting.

Result in 40 games from 3moves_GM_04.epd EPD opening 3-mover suite, which disfavors Lc0, because it's not a PGN opening suite to fill Lc0 history planes. Positions are random, balanced, from GM games. Time control is 60''+1''.

Code: Select all

Games Completed = 40 of 40 (Avg game length = 144.713 sec)
Settings = Gauntlet/128MB/60000ms+1000ms/M 150cp for 2 moves, D 35 moves/EPD:C:\LittleBlitzer\3moves_GM_04.epd(817)
Time = 5900 sec elapsed, 0 sec remaining
 1.  SF_dev 4 cores            	18.5/40	5-8-27  	(L: m=0 t=0 i=0 a=8)	(D: r=0 i=0 f=0 s=0 a=27)	(tpm=2341.3 d=23.37 nps=6118319)
 2.  lc0_v18.1_ID11261        	21.5/40	8-5-27  	(L: m=0 t=0 i=0 a=5)	(D: r=0 i=0 f=0 s=0 a=27)	(tpm=2152.9 d=8.28 nps=2958)
 

Lc0 wins +8 -5 =27.

Old sentence but it is true now : Every chess games continue up to the last move.
So based on experiments of you and others a well-founded declaration is in openings and middle game LC0 is positionally stronger than Stockfish in some measure.
Naturally an NN engine what was trained from start position with self playing much more stronger in openings than in endings. The better positional and weaker tactical play is assignable to the using of MCTS. So if you use for start positions tactical (gambits) openings I have doubt about any benefit of LC0.
For a really stronger NN engine we need a well trained bigger NN.
A bigger NN needs stronger hardware so it is the question to be fair or not to be fair.

[Uri Blass]

On my fair price-wise hardware in openings and middlegames.

My GPU is GTX 1060 6GB and CPU is 4 core i7 4790 at 3.8 GHz.
Lc0 18.1 ID11261 is using 2 CPU threads and GPU
Stockfish_dev is using 4 CPU cores (4 threads).

Seeing how badly Lc0 underperforms in endgames (more than 350 Elo points weaker than the general play)
http://talkchess.com/forum3/viewtopic.p ... 9&start=33 ,
I used special adjudication rules to avoid endgames, and to see which engine achieves more often a decisive advantage from early openings all through to late middlegames.

• Adjudication:
  
  As Win if both engines show 150cp or more advantage for one engine for 2 consecutive moves. It corresponds to SF_dev Win rate of 90% in openings and middlegames (endgames need a bit higher cp value for 90% Win rate).
  
  As sudden Draw at move 35, so most endgames don't enter the game.

SF_dev uses Contempt=0, as with non-zero contempt it shows fake value of eval. Lc0 has no contempt setting.

Result in 40 games from 3moves_GM_04.epd EPD opening 3-mover suite, which disfavors Lc0, because it's not a PGN opening suite to fill Lc0 history planes. Positions are random, balanced, from GM games. Time control is 60''+1''.

Code: Select all

Games Completed = 40 of 40 (Avg game length = 144.713 sec)
Settings = Gauntlet/128MB/60000ms+1000ms/M 150cp for 2 moves, D 35 moves/EPD:C:\LittleBlitzer\3moves_GM_04.epd(817)
Time = 5900 sec elapsed, 0 sec remaining
 1.  SF_dev 4 cores            	18.5/40	5-8-27  	(L: m=0 t=0 i=0 a=8)	(D: r=0 i=0 f=0 s=0 a=27)	(tpm=2341.3 d=23.37 nps=6118319)
 2.  lc0_v18.1_ID11261        	21.5/40	8-5-27  	(L: m=0 t=0 i=0 a=5)	(D: r=0 i=0 f=0 s=0 a=27)	(tpm=2152.9 d=8.28 nps=2958)
 

Lc0 wins +8 -5 =27.

The next match is against Komodo 12.1.1.

I find your adjudication to be unfair when you use evaluation of engines.
If I understand correctly your adjudication rules you can adjudicate at move 35 as draw in case that stockfish see a decisive advantage for itself only because lc0 does not see it.

If you really want to test only opening and middle game then the right solution is simply to replace lc0 by stockfish when you go from the middle game to the endgame with no adjudication.

[jp]

I find your adjudication to be unfair when you use evaluation of engines.
If I understand correctly your adjudication rules you can adjudicate at move 35 as draw in case that stockfish see a decisive advantage for itself only because lc0 does not see it.

If you really want to test only opening and middle game then the right solution is simply to replace lc0 by stockfish when you go from the middle game to the endgame with no adjudication.

Yep, I agree that Uri's solution is better, because lc0 always seems to think it has an advantage.

[Milos]

SF_dev uses Contempt=0, as with non-zero contempt it shows fake value of eval. Lc0 has no contempt setting.

Lol, Lc0 behaves as it has constant 100cp contempt.
When will ppl understand that it's not that much about endgames (Lc0 is certainly weak in them) but in fake eval. Lc0 always believes it's better than its opponent and converts only few of those "advantages". With perfect endgame it would convert a few more, but still far less than what its fake evaluation indicates.
This test of yours test is probably the worst you have ever produced. It has 0 actual merit.

[MikeGL]

On my fair price-wise hardware in openings and middlegames.

My GPU is GTX 1060 6GB and CPU is 4 core i7 4790 at 3.8 GHz.
Lc0 18.1 ID11261 is using 2 CPU threads and GPU
Stockfish_dev is using 4 CPU cores (4 threads).

Seeing how badly Lc0 underperforms in endgames (more than 350 Elo points weaker than the general play)
http://talkchess.com/forum3/viewtopic.p ... 9&start=33 ,
I used special adjudication rules to avoid endgames, and to see which engine achieves more often a decisive advantage from early openings all through to late middlegames.

• Adjudication:
  
  As Win if both engines show 150cp or more advantage for one engine for 2 consecutive moves. It corresponds to SF_dev Win rate of 90% in openings and middlegames (endgames need a bit higher cp value for 90% Win rate).
  
  As sudden Draw at move 35, so most endgames don't enter the game.

SF_dev uses Contempt=0, as with non-zero contempt it shows fake value of eval. Lc0 has no contempt setting.

Result in 40 games from 3moves_GM_04.epd EPD opening 3-mover suite, which disfavors Lc0, because it's not a PGN opening suite to fill Lc0 history planes. Positions are random, balanced, from GM games. Time control is 60''+1''.

Code: Select all

Games Completed = 40 of 40 (Avg game length = 144.713 sec)
Settings = Gauntlet/128MB/60000ms+1000ms/M 150cp for 2 moves, D 35 moves/EPD:C:\LittleBlitzer\3moves_GM_04.epd(817)
Time = 5900 sec elapsed, 0 sec remaining
 1.  SF_dev 4 cores            	18.5/40	5-8-27  	(L: m=0 t=0 i=0 a=8)	(D: r=0 i=0 f=0 s=0 a=27)	(tpm=2341.3 d=23.37 nps=6118319)
 2.  lc0_v18.1_ID11261        	21.5/40	8-5-27  	(L: m=0 t=0 i=0 a=5)	(D: r=0 i=0 f=0 s=0 a=27)	(tpm=2152.9 d=8.28 nps=2958)
 

Lc0 wins +8 -5 =27.

The next match is against Komodo 12.1.1.

Would be be nice to review this PGN games, I might pick up a strategic lessons or two.
Thanks in advance.

[dkappe]

You may want to use this tool to see how your gpu and cpu compare.

https://github.com/dkappe/leela-ratio

[Laskos]

On my fair price-wise hardware in openings and middlegames.

My GPU is GTX 1060 6GB and CPU is 4 core i7 4790 at 3.8 GHz.
Lc0 18.1 ID11261 is using 2 CPU threads and GPU
Stockfish_dev is using 4 CPU cores (4 threads).

Seeing how badly Lc0 underperforms in endgames (more than 350 Elo points weaker than the general play)
http://talkchess.com/forum3/viewtopic.p ... 9&start=33 ,
I used special adjudication rules to avoid endgames, and to see which engine achieves more often a decisive advantage from early openings all through to late middlegames.

• Adjudication:
  
  As Win if both engines show 150cp or more advantage for one engine for 2 consecutive moves. It corresponds to SF_dev Win rate of 90% in openings and middlegames (endgames need a bit higher cp value for 90% Win rate).
  
  As sudden Draw at move 35, so most endgames don't enter the game.

SF_dev uses Contempt=0, as with non-zero contempt it shows fake value of eval. Lc0 has no contempt setting.

Result in 40 games from 3moves_GM_04.epd EPD opening 3-mover suite, which disfavors Lc0, because it's not a PGN opening suite to fill Lc0 history planes. Positions are random, balanced, from GM games. Time control is 60''+1''.

Code: Select all

Games Completed = 40 of 40 (Avg game length = 144.713 sec)
Settings = Gauntlet/128MB/60000ms+1000ms/M 150cp for 2 moves, D 35 moves/EPD:C:\LittleBlitzer\3moves_GM_04.epd(817)
Time = 5900 sec elapsed, 0 sec remaining
 1.  SF_dev 4 cores            	18.5/40	5-8-27  	(L: m=0 t=0 i=0 a=8)	(D: r=0 i=0 f=0 s=0 a=27)	(tpm=2341.3 d=23.37 nps=6118319)
 2.  lc0_v18.1_ID11261        	21.5/40	8-5-27  	(L: m=0 t=0 i=0 a=5)	(D: r=0 i=0 f=0 s=0 a=27)	(tpm=2152.9 d=8.28 nps=2958)
 

Lc0 wins +8 -5 =27.

The next match is against Komodo 12.1.1.

I find your adjudication to be unfair when you use evaluation of engines.
If I understand correctly your adjudication rules you can adjudicate at move 35 as draw in case that stockfish see a decisive advantage for itself only because lc0 does not see it.

If you really want to test only opening and middle game then the right solution is simply to replace lc0 by stockfish when you go from the middle game to the endgame with no adjudication.

Fair enough, it's just a bit tedious, but I did that now. Those 8 "losses" of SF were real "losses" where SF considered itself at 150cp disadvantage, so unless draw rate would be very low, they are significant.

Anyway, I re-did the test:

SF-dev vs Lc0 up to move 25(28), no adjudications as wins/losses, as the EPD openings are already 3-movers. 35(38) was a bit too much, many games went well into endgames.

Then, from move 25(28) a let Komodo 12.1.1 Syzygy (enabled with 6-men) in self-games to continue them to the end. Komodo in endgames is as strong as SF, but I wanted a different engine because SF might be biased in ending positions of games played by SF.

The result was:

+6 -6 =28 in 40 games at 1'+1''.

So, 25 moves of opening + middlegame of Lc0 against SF-dev (4 cores) resulted in fairly equal positions. As I played the games in Cutechess GUI, I saw an interesting phenomenon: more games started with Komodo agreeing that SF position at move 25(28) is better, only for the final result to be equal in 40 games.

[Laskos]

SF_dev uses Contempt=0, as with non-zero contempt it shows fake value of eval. Lc0 has no contempt setting.

Lol, Lc0 behaves as it has constant 100cp contempt.
When will ppl understand that it's not that much about endgames (Lc0 is certainly weak in them) but in fake eval. Lc0 always believes it's better than its opponent and converts only few of those "advantages". With perfect endgame it would convert a few more, but still far less than what its fake evaluation indicates.
This test of yours test is probably the worst you have ever produced. It has 0 actual merit.

Right, but it gives some insights as starting point, and the "enhanced result" (see above post) is not that different. To move 25(28), SF_dev and Lc0 end in fairly equal positions, although a regular engine like Komodo more often considered SF position as better.