leela is official(?) better than sf9

[ernest]

Hi Javier,
Just a (stupid?) question :
on the diagram, I see the 1st game (here game 82).
How do I see the other games ?

Maybe it's a problem of my iPad Safari browser ?

[tpoppins]

Above the board there's a drop-down box with a list of the games. Click on it to open the list.

[ernest]

Above the board there's a drop-down box with a list of the games. Click on it to open the list.

The one marked "..." ! Thanks !

[Laskos]

It is very instructive to examine the debug output of the engine lc0.

I am giving here an example calculated in my computer but in the following link appears another similar debug output

https://gist.github.com/zz4032/7af71a69 ... a3d27cb0bb

The following output has been obtained with Cutechess doing a tournament between lc0 11261 and Fire 7 with time control 5 minutes and 3 sec starting from the initial position. (GPU GTX 1070 and CPU i7 7700H)

At the first line I highlight time 38 milliseconds, nodes 4 and hashfull 0. For being at the start of the game the nodes have started from 0 (the 4 nodes correspond to the 38 millisecs) and the hash table is empty.

<lc011261(0): info depth 1 seldepth 2 time 38 nodes 4 score cp 16 hashfull 0 nps 105 tbhits 0 pv g1f3 g8f6
<lc011261(0): info depth 1 seldepth 2 time 53 nodes 9 score cp 21 hashfull 0 nps 169 tbhits 0 pv e2e4 c7c5
<lc011261(0): info depth 2 seldepth 3 time 69 nodes 27 score cp -1 hashfull 0 nps 391 tbhits 0 pv e2e3 g8f6 g1f3
<lc011261(0): info depth 2 seldepth 3 time 85 nodes 50 score cp 7 hashfull 0 nps 588 tbhits 0 pv c2c4 c7c5 e2e3
<lc011261(0): info depth 2 seldepth 4 time 101 nodes 77 score cp 9 hashfull 0 nps 762 tbhits 0 pv c2c4 c7c5 e2e3 b8a6
<lc011261(0): info depth 3 seldepth 4 time 131 nodes 157 score cp 12 hashfull 0 nps 1198 tbhits 0 pv c2c4 c7c5 g1f3 g8f6
<lc011261(0): info depth 3 seldepth 5 time 147 nodes 185 score cp 13 hashfull 0 nps 1258 tbhits 0 pv c2c4 c7c5 g1f3 g8f6
<lc011261(0): info depth 3 seldepth 5 time 191 nodes 279 score cp 6 hashfull 1 nps 1460 tbhits 0 pv d2d4 g8f6 c2c4 e7e6 g1f3 b8a6
<lc011261(0): info depth 3 seldepth 6 time 248 nodes 466 score cp 10 hashfull 2 nps 1879 tbhits 0 pv d2d4 g8f6 g1f3 e7e6 c2c4 d7d5
<lc011261(0): info depth 3 seldepth 6 time 287 nodes 584 score cp 13 hashfull 2 nps 2034 tbhits 0 pv c2c4 c7c5 e2e3 g8f6 g1f3
<lc011261(0): info depth 4 seldepth 7 time 320 nodes 680 score cp 13 hashfull 3 nps 2125 tbhits 0 pv d2d4 d7d5 c2c4 e7e6 g1f3 g8f6
<lc011261(0): info depth 4 seldepth 7 time 360 nodes 796 score cp 19 hashfull 3 nps 2211 tbhits 0 pv e2e4 c7c5 g1f3 b8c6 b1c3 g8f6 a1b1
<lc011261(0): info depth 4 seldepth 8 time 400 nodes 924 score cp 19 hashfull 4 nps 2310 tbhits 0 pv e2e4 c7c5 g1f3 b8c6 b1c3 g8f6 d2d4
<lc011261(0): info depth 4 seldepth 9 time 503 nodes 1281 score cp 23 hashfull 6 nps 2546 tbhits 0 pv e2e4 c7c5 g1f3 b8c6 b1c3 g8f6 d2d4
<lc011261(0): info depth 4 seldepth 10 time 590 nodes 1571 score cp 26 hashfull 7 nps 2662 tbhits 0 pv e2e4 c7c5 g1f3 b8c6 b1c3 g8f6 d2d4 a8b8
<lc011261(0): info depth 5 seldepth 10 time 976 nodes 3026 score cp 25 hashfull 13 nps 3100 tbhits 0 pv e2e4 c7c5 g1f3 b8c6 b1c3 g8f6 d2d4 c5d4
<lc011261(0): info depth 5 seldepth 11 time 1431 nodes 4819 score cp 22 hashfull 20 nps 3367 tbhits 0 pv e2e4 c7c5 g1f3 b8c6 b1c3 g8f6 d2d4 c5d4
<lc011261(0): info depth 5 seldepth 12 time 1879 nodes 6611 score cp 19 hashfull 27 nps 3518 tbhits 0 pv e2e4 c7c5 g1f3 b8c6 b1c3 g8f6 d2d4 c5d4
<lc011261(0): info depth 5 seldepth 13 time 2506 nodes 9171 score cp 17 hashfull 37 nps 3659 tbhits 0 pv e2e4 c7c5 g1f3 b8c6 b1c3 g8f6 d2d4 c5d4 a1b1
<lc011261(0): info depth 5 seldepth 14 time 3289 nodes 12499 score cp 14 hashfull 50 nps 3800 tbhits 0 pv e2e4 c7c5 g1f3 d7d6 b1c3 g8f6 d2d4 c5d4 f3d4 g7g6
<lc011261(0): info depth 5 seldepth 15 time 4103 nodes 16083 score cp 15 hashfull 63 nps 3919 tbhits 0 pv e2e4 c7c5 g1f3 d7d6 b1c3 g8f6 d2d4 c5d4 f3d4 g7g6 f1b5
<lc011261(0): info depth 5 seldepth 16 time 4345 nodes 17107 score cp 16 hashfull 67 nps 3937 tbhits 0 pv e2e4 c7c5 g1f3 d7d6 b1c3 g8f6 d2d4 c5d4 f3d4 g7g6 f1b5 c8d7
<lc011261(0): info depth 6 seldepth 16 time 5403 nodes 21433 score cp 19 hashfull 83 nps 3966 tbhits 0 pv e2e4 e7e6 d2d4 d7d5 b1c3 g8f6 e4e5 f6d7 f2f4 c7c5 g1f3 b8c6 c1e3
<lc011261(0): info depth 6 seldepth 17 time 5592 nodes 22202 score cp 19 hashfull 86 nps 3970 tbhits 0 pv e2e4 e7e6 d2d4 d7d5 b1c3 g8f6 e4e5 f6d7 f2f4 c7c5 g1f3 b8c6 c1e3
<lc011261(0): info depth 6 seldepth 18 time 6905 nodes 27839 score cp 20 hashfull 108 nps 4031 tbhits 0 pv e2e4 c7c5 g1f3 b8c6 b1c3 g7g6 d2d4 c5d4 f3d4 f8g7 c1e3 g8f6 f1e2
<lc011261(0): info depth 6 seldepth 19 time 8312 nodes 33987 score cp 20 hashfull 130 nps 4088 tbhits 0 pv e2e4 c7c5 g1f3 d7d6 b1c3 g8f6 d2d4 c5d4 f3d4 g7g6 f1b5 c8d7 b5d7
<lc011261(0): info depth 6 seldepth 19 time 12624 nodes 52956 score cp 18 hashfull 199 nps 4194 tbhits 0 pv e2e4 c7c5 g1f3 d7d6 b1c3 g8f6 d2d4 c5d4 f3d4 g7g6 f1b5 c8d7 b5d7
<lc011261(0): bestmove e2e4 ponder c7c5

lc0 finished the calculation at depth ply 6 and selective depth 19, with hasfull 199 which means 19.9 % of the hash tables filled and played 1.e4


>Fire_7(1): position startpos moves e2e4
>Fire_7(1): isready
<Fire_7(1): readyok
>Fire_7(1): go wtime 290376 btime 300000 winc 3000 binc 3000
<Fire_7(1): info time 114 nodes 320890 nps 2814824 tbhits 0 depth 8 multipv 1 score cp -32 pv e7e5 g1f3 b8c6 b1c3 f8c5 f1b5 c6d4 b5a4
...
<Fire_7(1): info time 9402 nodes 27172080 nps 2890031 tbhits 0 depth 16 multipv 1 score cp -12 pv e7e6 d2d4 d7d5 b1c3 g8f6 c1g5 f8e7 e4e5 f6d7 h2h4 h7h6 g5e3 c7c5 d1g4 e8f8 g1f3
<Fire_7(1): bestmove e7e6 ponder d2d4
>lc011261(0): position startpos moves e2e4 e7e6
>lc011261(0): isready
<lc011261(0): readyok
>lc011261(0): go wtime 290376 btime 293599 winc 3000 binc 3000

After the moves 1.e4 e6, lc0 starts from time 0, nodes 8859 and hashfull 200. The amount of nodes doesn´t start from 0 because lc0 uses the information stored in the hash table and continues from that point.
lc0 started the calculation at depth ply 4 and selective depth 4, less than depth ply 6/19 this happened because the move e6 of Fire was not guessed by lc0, though e6 was analized at depth 6 seldepth 16 and depth 6 seldepth 17.


<lc011261(0): info depth 4 seldepth 4 time 0 nodes 8859 score cp 18 hashfull 200 nps 0 tbhits 0 pv d2d4 d7d5 b1c3 g8f6 e4e5 f6d7 f2f4 c7c5 g1f3 b8c6 c1e3 d8b6 c3a4
<lc011261(0): info depth 6 seldepth 8 time 0 nodes 8860 score cp 18 hashfull 200 nps 0 tbhits 0 pv d2d4 d7d5 b1c3 g8f6 e4e5 f6d7 f2f4 c7c5 g1f3 b8c6 c1e3 d8b6 c3a4
<lc011261(0): info depth 6 seldepth 13 time 71 nodes 9116 score cp 18 hashfull 201 nps 3633 tbhits 0 pv d2d4 d7d5 b1c3 g8f6 e4e5 f6d7 f2f4 c7c5 g1f3 b8c6 c1e3 d8b6 c3a4
<lc011261(0): info depth 6 seldepth 14 time 434 nodes 10549 score cp 21 hashfull 207 nps 3896 tbhits 0 pv d2d4 d7d5 b1c3 g8f6 e4e5 f6d7 c3e2 c7c5 c2c3 b8c6 g1f3 f7f6 e2f4 d8e7
<lc011261(0): info depth 6 seldepth 15 time 2639 nodes 20350 score cp 19 hashfull 242 nps 4354 tbhits 0 pv d2d4 d7d5 e4d5 e6d5 g1f3 g8f6 c2c4 f8b4 b1c3 e8g8 f1e2 c7c5 e1g1
<lc011261(0): info depth 7 seldepth 15 time 4942 nodes 30339 score cp 21 hashfull 279 nps 4346 tbhits 0 pv d2d4 d7d5 e4d5 e6d5 g1f3 g8f6 c2c4 f8b4 b1c3 e8g8 f1e2 c7c5 e1g1
<lc011261(0): info depth 7 seldepth 16 time 6980 nodes 39314 score cp 20 hashfull 311 nps 4363 tbhits 0 pv d2d4 d7d5 e4d5 e6d5 g1f3 g8f6 c2c4 f8b4 b1c3 e8g8 f1e2 c7c5 e1g1
<lc011261(0): info depth 7 seldepth 16 time 11999 nodes 62871 score cp 19 hashfull 390 nps 4501 tbhits 0 pv d2d4 d7d5 e4d5 e6d5 g1f3 g8f6 c2c4 f8b4 b1c3 e8g8 f1e2 c7c5 e1g1 c5d4 f3d4
<lc011261(0): info depth 7 seldepth 17 time 14464 nodes 74413 score cp 18 hashfull 429 nps 4532 tbhits 0 pv d2d4 d7d5 e4d5 e6d5 g1f3 g8f6 c2c4 f8b4 b1c3 e8g8 f1e2 c7c5 e1g1 c5d4 f3d4
<lc011261(0): info depth 7 seldepth 17 time 17552 nodes 88264 score cp 18 hashfull 478 nps 4524 tbhits 0 pv d2d4 d7d5 e4d5 e6d5 g1f3 g8f6 c2c4 f8b4 b1c3 e8g8 f1e2 c7c5 e1g1 c5d4 f3d4 b8c6 c4d5
<lc011261(0): bestmove d2d4 ponder d7d5

lc0 finished the calculation at depth ply 7 and selective depth 17

>Fire_7(1): position startpos moves e2e4 e7e6 d2d4
>Fire_7(1): isready
<Fire_7(1): readyok
>Fire_7(1): go wtime 275824 btime 293599 winc 3000 binc 3000
<Fire_7(1): info time 2 nodes 4356 nps 2178000 tbhits 0 depth 8 multipv 1 score cp -12 pv d7d5 b1c3 g8f6 c1g5 f8e7 e4e5 f6d7 h2h4
...
<Fire_7(1): info time 13302 nodes 39339735 nps 2957430 tbhits 0 depth 17 multipv 1 score cp -20 pv d7d5 b1d2 g8f6 f1d3 c7c5 e4e5 f6d7 c2c3 b8c6 g1e2 f8e7 d2f3 b7b6 e1g1 c8b7 f1e1 h7h6
<Fire_7(1): bestmove d7d5 ponder b1d2
>lc011261(0): position startpos moves e2e4 e7e6 d2d4 d7d5

After 1.e4 e6 2.d4 d5, lc0 at time 66 millisecs, starts with nodes 27821 and hashfull 480. The amount of nodes doesn´t start from 0 again.
lc0 started the calculation at depth ply 8 and selective depth 14 continuing from the depth 7/17 reached in the previous move probably due to that the move d5 of Fire was guessed by lc0. During the computation the depth varied from 7 to 8 several times

>lc011261(0): isready
<lc011261(0): readyok
>lc011261(0): go wtime 275824 btime 283299 winc 3000 binc 3000
<lc011261(0): info depth 8 seldepth 14 time 66 nodes 27821 score cp 20 hashfull 480 nps 3878 tbhits 0 pv e4d5 e6d5 g1f3 g8f6 c2c4 f8b4 b1c3 e8g8 f1e2 c7c5 e1g1 c5d4 f3d4 b8c6 c4d5
<lc011261(0): info depth 7 seldepth 14 time 188 nodes 28333 score cp 20 hashfull 482 nps 4085 tbhits 0 pv e4d5 e6d5 g1f3 g8f6 c2c4 f8b4 b1c3 e8g8 f1e2 c7c5 e1g1 c5d4 f3d4 b8c6 c4d5
<lc011261(0): info depth 7 seldepth 15 time 773 nodes 30893 score cp 20 hashfull 491 nps 4305 tbhits 0 pv e4d5 e6d5 g1f3 g8f6 c2c4 f8b4 b1c3 e8g8 f1e2 c7c5 e1g1 c5d4 f3d4 b8c6 c4d5
<lc011261(0): info depth 8 seldepth 15 time 997 nodes 31919 score cp 20 hashfull 495 nps 4367 tbhits 0 pv e4d5 e6d5 g1f3 g8f6 c2c4 f8b4 b1c3 e8g8 f1e2 c7c5 e1g1 c5d4 f3d4 b8c6 c4d5
<lc011261(0): info depth 7 seldepth 15 time 1062 nodes 32175 score cp 20 hashfull 496 nps 4340 tbhits 0 pv e4d5 e6d5 g1f3 g8f6 c2c4 f8b4 b1c3 e8g8 f1e2 c7c5 e1g1 c5d4 f3d4 b8c6 c4d5
<lc011261(0): info depth 8 seldepth 15 time 1182 nodes 32687 score cp 19 hashfull 498 nps 4333 tbhits 0 pv e4d5 e6d5 g1f3 g8f6 c2c4 f8b4 b1c3 e8g8 f1e2 c7c5 e1g1 c5d4 f3d4 b8c6 c4d5
<lc011261(0): info depth 7 seldepth 15 time 1310 nodes 33199 score cp 19 hashfull 500 nps 4300 tbhits 0 pv e4d5 e6d5 g1f3 g8f6 c2c4 f8b4 b1c3 e8g8 f1e2 c7c5 e1g1 c5d4 f3d4 b8c6 c4d5
<lc011261(0): info depth 8 seldepth 15 time 1378 nodes 33457 score cp 19 hashfull 501 nps 4275 tbhits 0 pv e4d5 e6d5 g1f3 g8f6 c2c4 f8b4 b1c3 e8g8 f1e2 c7c5 e1g1 c5d4 f3d4 b8c6 c4d5
<lc011261(0): info depth 7 seldepth 15 time 1436 nodes 33713 score cp 19 hashfull 502 nps 4281 tbhits 0 pv e4d5 e6d5 g1f3 g8f6 c2c4 f8b4 b1c3 e8g8 f1e2 c7c5 e1g1 c5d4 f3d4 b8c6 c4d5
<lc011261(0): info depth 8 seldepth 15 time 1486 nodes 33969 score cp 19 hashfull 503 nps 4309 tbhits 0 pv e4d5 e6d5 g1f3 g8f6 c2c4 f8b4 b1c3 e8g8 f1e2 b4c3 b2c3 d5c4 e2c4 f8e8 c1e3
<lc011261(0): info depth 8 seldepth 16 time 2761 nodes 39354 score cp 19 hashfull 523 nps 4269 tbhits 0 pv e4d5 e6d5 g1f3 g8f6 c2c4 f8b4 b1c3 e8g8 f1e2 b4c3 b2c3 d5c4 e2c4 f8e8 c1e3
<lc011261(0): info depth 8 seldepth 16 time 7779 nodes 61575 score cp 18 hashfull 604 nps 4372 tbhits 0 pv e4d5 e6d5 g1f3 g8f6 c2c4 c8e6 b1c3 d5c4 f1e2 b8c6 e1g1 f8e7 d1a4 e8g8
<lc011261(0): info depth 8 seldepth 17 time 12176 nodes 80180 score cp 17 hashfull 674 nps 4321 tbhits 0 pv e4d5 e6d5 g1f3 g8f6 c2c4 c8e6 b1c3 d5c4 f3g5 e6d5 c3d5 d8d5 f1e2 f8b4 e1f1
<lc011261(0): info depth 8 seldepth 17 time 16601 nodes 98634 score cp 17 hashfull 745 nps 4281 tbhits 0 pv e4d5 e6d5 g1f3 g8f6 c2c4 c8e6 b1c3 d5c4 d1a4 c7c6 f1c4 b7b5 c3b5 c6b5 c4b5
<lc011261(0): bestmove e4d5 ponder e6d5

Yes, you are right, I can reproduce this output. So, it seems we usually have nothing wrong in these matches. But the choice of openings is crucial, I didn't quite expected that. I think that my statement that DeepMind team used openings which are very convenient for A0 is valid. I tested at 2 minutes + 2 seconds increment time control. Lc0 v17 ID11261 was on 2 threads and GTX 1060, SF8 was on 4 i7 3.8 GHz cores.

From those 12 A0 opening positions used in 1200 games match of A0, I get side and reversed:
Score of lc0_v17 11261 vs SF 8: 4 - 4 - 16 [0.500]
Elo difference: 0.00 +/- 81.61

24 of 24 games finished.


Using 8-mover PGN opening book of Adam Hair, with history planes of Lc0 filled up, I get:
Score of lc0_v17 11261 vs SF 8: 0 - 12 - 28 [0.350]
Elo difference: -107.54 +/- 54.71

40 of 40 games finished.


Our differences probably come from using different set of openings, as Lc0 and A0 seem sensitive to the choice. 12 openings for 1200 games match and 1 opening for 100 games match is bad testing methodology by A0 team. And probably on purpose bad, as it bolster A0 (or Lc0) by about 100 Elo points.

[corres]

AB engines are tested mainly for middle game.
NN engines play learning games from start position so their knowledge decreases going further from start
position. These facts cause that starting the games from a sort opening book or without opening book is favorable for NN engines and disadvantageous for AB engines.

[chrisw]

AB engines are tested mainly for middle game.
NN engines play learning games from start position so their knowledge decreases going further from start
position. These facts cause that starting the games from a sort opening book or without opening book is favorable for NN engines and disadvantageous for AB engines.

The fewer pieces there are, especially pieces that can combine in mating nets, the stronger is the AB search (the search width is naturally smaller, AB is less likely to prune away important lines. and, the opportunities to create imbalances are fewer, if there is an imbalance, it is probably already there). NN-MCTS is better at finding imbalances and potential mating net structures, the opportunity for that is early on.

Your point of greater learnt knowledge at start (eg most pieces still on board) is also true. What's also true is that there is no reason why more training and larger nets can't overcome the weaknesses in later parts of the game. Just give it training time.

[chrisw]

It is very instructive to examine the debug output of the engine lc0.

I am giving here an example calculated in my computer but in the following link appears another similar debug output

https://gist.github.com/zz4032/7af71a69 ... a3d27cb0bb

The following output has been obtained with Cutechess doing a tournament between lc0 11261 and Fire 7 with time control 5 minutes and 3 sec starting from the initial position. (GPU GTX 1070 and CPU i7 7700H)

At the first line I highlight time 38 milliseconds, nodes 4 and hashfull 0. For being at the start of the game the nodes have started from 0 (the 4 nodes correspond to the 38 millisecs) and the hash table is empty.

<lc011261(0): info depth 1 seldepth 2 time 38 nodes 4 score cp 16 hashfull 0 nps 105 tbhits 0 pv g1f3 g8f6
<lc011261(0): info depth 1 seldepth 2 time 53 nodes 9 score cp 21 hashfull 0 nps 169 tbhits 0 pv e2e4 c7c5
<lc011261(0): info depth 2 seldepth 3 time 69 nodes 27 score cp -1 hashfull 0 nps 391 tbhits 0 pv e2e3 g8f6 g1f3
<lc011261(0): info depth 2 seldepth 3 time 85 nodes 50 score cp 7 hashfull 0 nps 588 tbhits 0 pv c2c4 c7c5 e2e3
<lc011261(0): info depth 2 seldepth 4 time 101 nodes 77 score cp 9 hashfull 0 nps 762 tbhits 0 pv c2c4 c7c5 e2e3 b8a6
<lc011261(0): info depth 3 seldepth 4 time 131 nodes 157 score cp 12 hashfull 0 nps 1198 tbhits 0 pv c2c4 c7c5 g1f3 g8f6
<lc011261(0): info depth 3 seldepth 5 time 147 nodes 185 score cp 13 hashfull 0 nps 1258 tbhits 0 pv c2c4 c7c5 g1f3 g8f6
<lc011261(0): info depth 3 seldepth 5 time 191 nodes 279 score cp 6 hashfull 1 nps 1460 tbhits 0 pv d2d4 g8f6 c2c4 e7e6 g1f3 b8a6
<lc011261(0): info depth 3 seldepth 6 time 248 nodes 466 score cp 10 hashfull 2 nps 1879 tbhits 0 pv d2d4 g8f6 g1f3 e7e6 c2c4 d7d5
<lc011261(0): info depth 3 seldepth 6 time 287 nodes 584 score cp 13 hashfull 2 nps 2034 tbhits 0 pv c2c4 c7c5 e2e3 g8f6 g1f3
<lc011261(0): info depth 4 seldepth 7 time 320 nodes 680 score cp 13 hashfull 3 nps 2125 tbhits 0 pv d2d4 d7d5 c2c4 e7e6 g1f3 g8f6
<lc011261(0): info depth 4 seldepth 7 time 360 nodes 796 score cp 19 hashfull 3 nps 2211 tbhits 0 pv e2e4 c7c5 g1f3 b8c6 b1c3 g8f6 a1b1
<lc011261(0): info depth 4 seldepth 8 time 400 nodes 924 score cp 19 hashfull 4 nps 2310 tbhits 0 pv e2e4 c7c5 g1f3 b8c6 b1c3 g8f6 d2d4
<lc011261(0): info depth 4 seldepth 9 time 503 nodes 1281 score cp 23 hashfull 6 nps 2546 tbhits 0 pv e2e4 c7c5 g1f3 b8c6 b1c3 g8f6 d2d4
<lc011261(0): info depth 4 seldepth 10 time 590 nodes 1571 score cp 26 hashfull 7 nps 2662 tbhits 0 pv e2e4 c7c5 g1f3 b8c6 b1c3 g8f6 d2d4 a8b8
<lc011261(0): info depth 5 seldepth 10 time 976 nodes 3026 score cp 25 hashfull 13 nps 3100 tbhits 0 pv e2e4 c7c5 g1f3 b8c6 b1c3 g8f6 d2d4 c5d4
<lc011261(0): info depth 5 seldepth 11 time 1431 nodes 4819 score cp 22 hashfull 20 nps 3367 tbhits 0 pv e2e4 c7c5 g1f3 b8c6 b1c3 g8f6 d2d4 c5d4
<lc011261(0): info depth 5 seldepth 12 time 1879 nodes 6611 score cp 19 hashfull 27 nps 3518 tbhits 0 pv e2e4 c7c5 g1f3 b8c6 b1c3 g8f6 d2d4 c5d4
<lc011261(0): info depth 5 seldepth 13 time 2506 nodes 9171 score cp 17 hashfull 37 nps 3659 tbhits 0 pv e2e4 c7c5 g1f3 b8c6 b1c3 g8f6 d2d4 c5d4 a1b1
<lc011261(0): info depth 5 seldepth 14 time 3289 nodes 12499 score cp 14 hashfull 50 nps 3800 tbhits 0 pv e2e4 c7c5 g1f3 d7d6 b1c3 g8f6 d2d4 c5d4 f3d4 g7g6
<lc011261(0): info depth 5 seldepth 15 time 4103 nodes 16083 score cp 15 hashfull 63 nps 3919 tbhits 0 pv e2e4 c7c5 g1f3 d7d6 b1c3 g8f6 d2d4 c5d4 f3d4 g7g6 f1b5
<lc011261(0): info depth 5 seldepth 16 time 4345 nodes 17107 score cp 16 hashfull 67 nps 3937 tbhits 0 pv e2e4 c7c5 g1f3 d7d6 b1c3 g8f6 d2d4 c5d4 f3d4 g7g6 f1b5 c8d7
<lc011261(0): info depth 6 seldepth 16 time 5403 nodes 21433 score cp 19 hashfull 83 nps 3966 tbhits 0 pv e2e4 e7e6 d2d4 d7d5 b1c3 g8f6 e4e5 f6d7 f2f4 c7c5 g1f3 b8c6 c1e3
<lc011261(0): info depth 6 seldepth 17 time 5592 nodes 22202 score cp 19 hashfull 86 nps 3970 tbhits 0 pv e2e4 e7e6 d2d4 d7d5 b1c3 g8f6 e4e5 f6d7 f2f4 c7c5 g1f3 b8c6 c1e3
<lc011261(0): info depth 6 seldepth 18 time 6905 nodes 27839 score cp 20 hashfull 108 nps 4031 tbhits 0 pv e2e4 c7c5 g1f3 b8c6 b1c3 g7g6 d2d4 c5d4 f3d4 f8g7 c1e3 g8f6 f1e2
<lc011261(0): info depth 6 seldepth 19 time 8312 nodes 33987 score cp 20 hashfull 130 nps 4088 tbhits 0 pv e2e4 c7c5 g1f3 d7d6 b1c3 g8f6 d2d4 c5d4 f3d4 g7g6 f1b5 c8d7 b5d7
<lc011261(0): info depth 6 seldepth 19 time 12624 nodes 52956 score cp 18 hashfull 199 nps 4194 tbhits 0 pv e2e4 c7c5 g1f3 d7d6 b1c3 g8f6 d2d4 c5d4 f3d4 g7g6 f1b5 c8d7 b5d7
<lc011261(0): bestmove e2e4 ponder c7c5

lc0 finished the calculation at depth ply 6 and selective depth 19, with hasfull 199 which means 19.9 % of the hash tables filled and played 1.e4


>Fire_7(1): position startpos moves e2e4
>Fire_7(1): isready
<Fire_7(1): readyok
>Fire_7(1): go wtime 290376 btime 300000 winc 3000 binc 3000
<Fire_7(1): info time 114 nodes 320890 nps 2814824 tbhits 0 depth 8 multipv 1 score cp -32 pv e7e5 g1f3 b8c6 b1c3 f8c5 f1b5 c6d4 b5a4
...
<Fire_7(1): info time 9402 nodes 27172080 nps 2890031 tbhits 0 depth 16 multipv 1 score cp -12 pv e7e6 d2d4 d7d5 b1c3 g8f6 c1g5 f8e7 e4e5 f6d7 h2h4 h7h6 g5e3 c7c5 d1g4 e8f8 g1f3
<Fire_7(1): bestmove e7e6 ponder d2d4
>lc011261(0): position startpos moves e2e4 e7e6
>lc011261(0): isready
<lc011261(0): readyok
>lc011261(0): go wtime 290376 btime 293599 winc 3000 binc 3000

After the moves 1.e4 e6, lc0 starts from time 0, nodes 8859 and hashfull 200. The amount of nodes doesn´t start from 0 because lc0 uses the information stored in the hash table and continues from that point.
lc0 started the calculation at depth ply 4 and selective depth 4, less than depth ply 6/19 this happened because the move e6 of Fire was not guessed by lc0, though e6 was analized at depth 6 seldepth 16 and depth 6 seldepth 17.


<lc011261(0): info depth 4 seldepth 4 time 0 nodes 8859 score cp 18 hashfull 200 nps 0 tbhits 0 pv d2d4 d7d5 b1c3 g8f6 e4e5 f6d7 f2f4 c7c5 g1f3 b8c6 c1e3 d8b6 c3a4
<lc011261(0): info depth 6 seldepth 8 time 0 nodes 8860 score cp 18 hashfull 200 nps 0 tbhits 0 pv d2d4 d7d5 b1c3 g8f6 e4e5 f6d7 f2f4 c7c5 g1f3 b8c6 c1e3 d8b6 c3a4
<lc011261(0): info depth 6 seldepth 13 time 71 nodes 9116 score cp 18 hashfull 201 nps 3633 tbhits 0 pv d2d4 d7d5 b1c3 g8f6 e4e5 f6d7 f2f4 c7c5 g1f3 b8c6 c1e3 d8b6 c3a4
<lc011261(0): info depth 6 seldepth 14 time 434 nodes 10549 score cp 21 hashfull 207 nps 3896 tbhits 0 pv d2d4 d7d5 b1c3 g8f6 e4e5 f6d7 c3e2 c7c5 c2c3 b8c6 g1f3 f7f6 e2f4 d8e7
<lc011261(0): info depth 6 seldepth 15 time 2639 nodes 20350 score cp 19 hashfull 242 nps 4354 tbhits 0 pv d2d4 d7d5 e4d5 e6d5 g1f3 g8f6 c2c4 f8b4 b1c3 e8g8 f1e2 c7c5 e1g1
<lc011261(0): info depth 7 seldepth 15 time 4942 nodes 30339 score cp 21 hashfull 279 nps 4346 tbhits 0 pv d2d4 d7d5 e4d5 e6d5 g1f3 g8f6 c2c4 f8b4 b1c3 e8g8 f1e2 c7c5 e1g1
<lc011261(0): info depth 7 seldepth 16 time 6980 nodes 39314 score cp 20 hashfull 311 nps 4363 tbhits 0 pv d2d4 d7d5 e4d5 e6d5 g1f3 g8f6 c2c4 f8b4 b1c3 e8g8 f1e2 c7c5 e1g1
<lc011261(0): info depth 7 seldepth 16 time 11999 nodes 62871 score cp 19 hashfull 390 nps 4501 tbhits 0 pv d2d4 d7d5 e4d5 e6d5 g1f3 g8f6 c2c4 f8b4 b1c3 e8g8 f1e2 c7c5 e1g1 c5d4 f3d4
<lc011261(0): info depth 7 seldepth 17 time 14464 nodes 74413 score cp 18 hashfull 429 nps 4532 tbhits 0 pv d2d4 d7d5 e4d5 e6d5 g1f3 g8f6 c2c4 f8b4 b1c3 e8g8 f1e2 c7c5 e1g1 c5d4 f3d4
<lc011261(0): info depth 7 seldepth 17 time 17552 nodes 88264 score cp 18 hashfull 478 nps 4524 tbhits 0 pv d2d4 d7d5 e4d5 e6d5 g1f3 g8f6 c2c4 f8b4 b1c3 e8g8 f1e2 c7c5 e1g1 c5d4 f3d4 b8c6 c4d5
<lc011261(0): bestmove d2d4 ponder d7d5

lc0 finished the calculation at depth ply 7 and selective depth 17

>Fire_7(1): position startpos moves e2e4 e7e6 d2d4
>Fire_7(1): isready
<Fire_7(1): readyok
>Fire_7(1): go wtime 275824 btime 293599 winc 3000 binc 3000
<Fire_7(1): info time 2 nodes 4356 nps 2178000 tbhits 0 depth 8 multipv 1 score cp -12 pv d7d5 b1c3 g8f6 c1g5 f8e7 e4e5 f6d7 h2h4
...
<Fire_7(1): info time 13302 nodes 39339735 nps 2957430 tbhits 0 depth 17 multipv 1 score cp -20 pv d7d5 b1d2 g8f6 f1d3 c7c5 e4e5 f6d7 c2c3 b8c6 g1e2 f8e7 d2f3 b7b6 e1g1 c8b7 f1e1 h7h6
<Fire_7(1): bestmove d7d5 ponder b1d2
>lc011261(0): position startpos moves e2e4 e7e6 d2d4 d7d5

After 1.e4 e6 2.d4 d5, lc0 at time 66 millisecs, starts with nodes 27821 and hashfull 480. The amount of nodes doesn´t start from 0 again.
lc0 started the calculation at depth ply 8 and selective depth 14 continuing from the depth 7/17 reached in the previous move probably due to that the move d5 of Fire was guessed by lc0. During the computation the depth varied from 7 to 8 several times

>lc011261(0): isready
<lc011261(0): readyok
>lc011261(0): go wtime 275824 btime 283299 winc 3000 binc 3000
<lc011261(0): info depth 8 seldepth 14 time 66 nodes 27821 score cp 20 hashfull 480 nps 3878 tbhits 0 pv e4d5 e6d5 g1f3 g8f6 c2c4 f8b4 b1c3 e8g8 f1e2 c7c5 e1g1 c5d4 f3d4 b8c6 c4d5
<lc011261(0): info depth 7 seldepth 14 time 188 nodes 28333 score cp 20 hashfull 482 nps 4085 tbhits 0 pv e4d5 e6d5 g1f3 g8f6 c2c4 f8b4 b1c3 e8g8 f1e2 c7c5 e1g1 c5d4 f3d4 b8c6 c4d5
<lc011261(0): info depth 7 seldepth 15 time 773 nodes 30893 score cp 20 hashfull 491 nps 4305 tbhits 0 pv e4d5 e6d5 g1f3 g8f6 c2c4 f8b4 b1c3 e8g8 f1e2 c7c5 e1g1 c5d4 f3d4 b8c6 c4d5
<lc011261(0): info depth 8 seldepth 15 time 997 nodes 31919 score cp 20 hashfull 495 nps 4367 tbhits 0 pv e4d5 e6d5 g1f3 g8f6 c2c4 f8b4 b1c3 e8g8 f1e2 c7c5 e1g1 c5d4 f3d4 b8c6 c4d5
<lc011261(0): info depth 7 seldepth 15 time 1062 nodes 32175 score cp 20 hashfull 496 nps 4340 tbhits 0 pv e4d5 e6d5 g1f3 g8f6 c2c4 f8b4 b1c3 e8g8 f1e2 c7c5 e1g1 c5d4 f3d4 b8c6 c4d5
<lc011261(0): info depth 8 seldepth 15 time 1182 nodes 32687 score cp 19 hashfull 498 nps 4333 tbhits 0 pv e4d5 e6d5 g1f3 g8f6 c2c4 f8b4 b1c3 e8g8 f1e2 c7c5 e1g1 c5d4 f3d4 b8c6 c4d5
<lc011261(0): info depth 7 seldepth 15 time 1310 nodes 33199 score cp 19 hashfull 500 nps 4300 tbhits 0 pv e4d5 e6d5 g1f3 g8f6 c2c4 f8b4 b1c3 e8g8 f1e2 c7c5 e1g1 c5d4 f3d4 b8c6 c4d5
<lc011261(0): info depth 8 seldepth 15 time 1378 nodes 33457 score cp 19 hashfull 501 nps 4275 tbhits 0 pv e4d5 e6d5 g1f3 g8f6 c2c4 f8b4 b1c3 e8g8 f1e2 c7c5 e1g1 c5d4 f3d4 b8c6 c4d5
<lc011261(0): info depth 7 seldepth 15 time 1436 nodes 33713 score cp 19 hashfull 502 nps 4281 tbhits 0 pv e4d5 e6d5 g1f3 g8f6 c2c4 f8b4 b1c3 e8g8 f1e2 c7c5 e1g1 c5d4 f3d4 b8c6 c4d5
<lc011261(0): info depth 8 seldepth 15 time 1486 nodes 33969 score cp 19 hashfull 503 nps 4309 tbhits 0 pv e4d5 e6d5 g1f3 g8f6 c2c4 f8b4 b1c3 e8g8 f1e2 b4c3 b2c3 d5c4 e2c4 f8e8 c1e3
<lc011261(0): info depth 8 seldepth 16 time 2761 nodes 39354 score cp 19 hashfull 523 nps 4269 tbhits 0 pv e4d5 e6d5 g1f3 g8f6 c2c4 f8b4 b1c3 e8g8 f1e2 b4c3 b2c3 d5c4 e2c4 f8e8 c1e3
<lc011261(0): info depth 8 seldepth 16 time 7779 nodes 61575 score cp 18 hashfull 604 nps 4372 tbhits 0 pv e4d5 e6d5 g1f3 g8f6 c2c4 c8e6 b1c3 d5c4 f1e2 b8c6 e1g1 f8e7 d1a4 e8g8
<lc011261(0): info depth 8 seldepth 17 time 12176 nodes 80180 score cp 17 hashfull 674 nps 4321 tbhits 0 pv e4d5 e6d5 g1f3 g8f6 c2c4 c8e6 b1c3 d5c4 f3g5 e6d5 c3d5 d8d5 f1e2 f8b4 e1f1
<lc011261(0): info depth 8 seldepth 17 time 16601 nodes 98634 score cp 17 hashfull 745 nps 4281 tbhits 0 pv e4d5 e6d5 g1f3 g8f6 c2c4 c8e6 b1c3 d5c4 d1a4 c7c6 f1c4 b7b5 c3b5 c6b5 c4b5
<lc011261(0): bestmove e4d5 ponder e6d5

Yes, you are right, I can reproduce this output. So, it seems we usually have nothing wrong in these matches. But the choice of openings is crucial, I didn't quite expected that. I think that my statement that DeepMind team used openings which are very convenient for A0 is valid. I tested at 2 minutes + 2 seconds increment time control. Lc0 v17 ID11261 was on 2 threads and GTX 1060, SF8 was on 4 i7 3.8 GHz cores.

From those 12 A0 opening positions used in 1200 games match of A0, I get side and reversed:
Score of lc0_v17 11261 vs SF 8: 4 - 4 - 16 [0.500]
Elo difference: 0.00 +/- 81.61

24 of 24 games finished.


Using 8-mover PGN opening book of Adam Hair, with history planes of Lc0 filled up, I get:
Score of lc0_v17 11261 vs SF 8: 0 - 12 - 28 [0.350]
Elo difference: -107.54 +/- 54.71

40 of 40 games finished.


Our differences probably come from using different set of openings, as Lc0 and A0 seem sensitive to the choice. 12 openings for 1200 games match and 1 opening for 100 games match is bad testing methodology by A0 team. And probably on purpose bad, as it bolster A0 (or Lc0) by about 100 Elo points.

Interesting, indeed. Critics are going to say "more games needed", especially so when your case is based on differential elo. However, what starts off one way, has a tendency to stay that way.

[Tobber]

Our differences probably come from using different set of openings, as Lc0 and A0 seem sensitive to the choice. 12 openings for 1200 games match and 1 opening for 100 games match is bad testing methodology by A0 team. And probably on purpose bad, as it bolster A0 (or Lc0) by about 100 Elo points.

Please read the paper and then check for yourself. The 1200 games match were selected as the 12 most popular openings from an online database.

/John

[Laskos]

Our differences probably come from using different set of openings, as Lc0 and A0 seem sensitive to the choice. 12 openings for 1200 games match and 1 opening for 100 games match is bad testing methodology by A0 team. And probably on purpose bad, as it bolster A0 (or Lc0) by about 100 Elo points.

Please read the paper and then check for yourself. The 1200 games match were selected as the 12 most popular openings from an online database.

/John

I don't understand what you are saying, and I won't read again the paper right now. How many positions they did use for 1200 games match? The fact is the 12 provided positions are very favorable for Lc0 and probably A0. Similarly, no openings at all heavily favors Lc0 and A0.

[Tobber]

Our differences probably come from using different set of openings, as Lc0 and A0 seem sensitive to the choice. 12 openings for 1200 games match and 1 opening for 100 games match is bad testing methodology by A0 team. And probably on purpose bad, as it bolster A0 (or Lc0) by about 100 Elo points.

Please read the paper and then check for yourself. The 1200 games match were selected as the 12 most popular openings from an online database.

/John

I don't understand what you are saying, and I won't read again the paper right now. How many positions they did use for 1200 games match? The fact is the 12 provided positions are very favorable for Lc0 and probably A0. Similarly, no openings at all heavily favors Lc0 and A0.

So the 12 most popular openings in human play favor Lc0 and A0? The idea was to show that A0 by self-play could figure out how to play the most popular openings. What is it you don't understand?
/John