Lc0 26.2 10x128 nets vs Stockfish 12

[MMarco]

Gauntlet: Lc0 26.2 10x128 nets vs Stockfish 12
Time control: 25s + 0.25s
Hardware: Ryzen 3750H, GTX 1660 Ti
Openings: Super GM 4 moves (500 pos.)
Bench (1 sec): Stockfish 12 = 1800 knps. Lc0 = 30 knps
Stockfish 12: 1 thread, hash=16, Move Overhead=0, abrok modern compile
Lc0 26.2 + 591226: minibatch-size=192, max-prefetch=86, syzygy-fast-play=false, move-overhead=0
Lc0 26.2 + other: minibatch-size=256, max-prefetch=115, syzygy-fast-play=false, move-overhead=0
MLH: moves-left-max-effect=0.1, moves-left-threshold=0.85, moves-left-slope=0.007, moves-left-quadratic-factor=1, moves-left-constant-factor=0, moves-left-scaled-factor=0
TBs and adj.: Syzygy 5-men, resign movecount=5 score=600, draw movenumber=35 movecount=5 score=5, time forfeit disabled
Games: https://gofile.io/d/l5TfkE
SV-nets: https://www.comp.nus.edu.sg/~sergio-v/new/128x10-t60-2/
jio-3450: https://cdn.discordapp.com/attachments/ ... 0000.pb.gz
tinker-6430: https://cdn.discordapp.com/attachments/ ... 0000.pb.gz

Code: Select all

   # PLAYER                  :  RATING  ERROR   PLAYED  (%)   CFS     W     D    L  D(%)
   1 Stockfish 12            :     0.0   ----    8000  54.1    99  1482  5689  829  71.1
   2 Lc0 26.2 + SV-5300      :   -13.7   11.2    1000  48.0    50   122   717  161  71.7
   3 Lc0 26.2 + SV-2990      :   -13.7   11.8    1000  48.0    79   114   733  153  73.3
   4 Lc0 26.2 + tinker-6430  :   -20.4   11.0    1000  47.1    89   116   710  174  71.0
   5 Lc0 26.2 + 591226       :   -30.2   10.4    1000  45.7    52    99   716  185  71.6
   6 Lc0 26.2 + jio-3450     :   -30.6   10.6    1000  45.6    79    93   727  180  72.7
   7 Lc0 26.2 + 703810 MLH   :   -37.3   12.1    1000  44.7    55   100   694  206  69.4
   8 Lc0 26.2 + 703810       :   -38.4   11.5    1000  44.5    82    94   703  203  70.3
   9 Lc0 26.2 + 721876       :   -45.5   10.7    1000  43.5   ---    91   689  220  68.9

White advantage = 11.44 +/- 2.11
Draw rate (equal opponents) = 72.22 % +/- 0.49

Depth stats:

Code: Select all

Engine                 Time/move  Depth   MIDG   EARLY    ENDG    LATE
Lc0 26.2 + 591226         0.57     9.49  10.28 | 11.05 |  8.77 |  7.77
Lc0 26.2 + 703810         0.58     9.48  10.08 | 10.74 |  9.01 |  7.84
Lc0 26.2 + 703810 MLH     0.58     9.55  10.11 | 10.78 |  9.17 |  7.84
Lc0 26.2 + 721876         0.59     9.34   9.83 | 10.63 |  8.87 |  7.90
Lc0 26.2 + SV-2990        0.58     9.19   9.99 | 10.05 |  8.48 |  7.64
Lc0 26.2 + SV-5300        0.57     9.11   9.83 | 10.35 |  8.40 |  7.71
Lc0 26.2 + jio-3450       0.57     9.15  10.00 | 10.77 |  8.45 |  7.36
Lc0 26.2 + tinker-6430    0.59     9.57  10.25 | 10.97 |  8.87 |  7.91
Stockfish 12              0.56    23.17  20.19 | 20.43 | 23.39 | 33.97

[Ozymandias]

Nicely done, any particular reason for the MLH parameters?

[MMarco]

Nicely done, any particular reason for the MLH parameters?

I picked these as they are currently used by a poster ("Trre") who did and published many tests on the T70 series in Lc0 discord. I assumed he must know about it.

[brianr]

Thank you for including one of my best nets (tinker-6430).
Unfortunately, the yaml and training logs are lost to the mists of time, but I'm fairly certain there was no MLH used, FWIW.
This net was trained using SL with 6.4M steps using T60 games and then finished with 30K RL steps.

[MMarco]

Thank you for including one of my best nets (tinker-6430).
Unfortunately, the yaml and training logs are lost to the mists of time, but I'm fairly certain there was no MLH used, FWIW.
This net was trained using SL with 6.4M steps using T60 games and then finished with 30K RL steps.

Thank you for training the net. Very impressive performance and among the best on the planet! Do you have any other 10b or 20b net that you would consider worthwhile testing? Larger than that my GPU suffers.

[brianr]

I could train another without the rule50 move counter which would be roughly about 20 Elo stronger, but it would take quite a lot of time (between one and three weeks). Before starting that effort, I'm trying to understand things a bit more.
If I cook another net that is significantly better I'll provide an update.

[MMarco]

Sounds good! 10x128 are still useful when one doesn't have an RTX card, or for Android devices.

[Ozymandias]

Or hyper-bullet games.