Fun experiment with NNUE dataset sizes

[MartinBryant]

I've recently begun working on an NNUE for Colossus. I'm using the Bullet trainer and the simple Chess768 architecture in its sample program. I've read posts that indicate the more the merrier when it comes to the number of positions to train networks on but I speculated that surely there were diminishing returns.

I built networks from datasets with 1 million positions, 2 million, etc up to 64 million and played a tournament. The searches were fixed depth of 9-ply to give the engines enough tactical ability to find mates. Results below. The '-N' on the end of the name is the number of millions.

Code: Select all

Rank Name                             Elo        +/-       nElo        +/-      Games      Score       Draw
   1 ColossusChess5-64             239.47       6.51     308.44       6.22      12000      79.9%      19.0%
   2 ColossusChess5-32             213.73       6.35     266.25       6.22      12000      77.4%      18.9%
   3 ColossusChess5-16             154.39       6.00     181.97       6.22      12000      70.9%      19.1%
   4 ColossusChess5-8               47.69       5.70      52.64       6.22      12000      56.8%      17.1%
   5 ColossusChess5-4              -68.66       5.91     -74.11       6.22      12000      40.2%      12.6%
   6 ColossusChess5-2             -235.40       7.34    -266.14       6.22      12000      20.5%       5.9%
   7 ColossusChess5-1             -538.80      14.26    -841.62       6.22      12000       4.3%       2.2%
Finished match

As expected the larger datasets produced the stronger networks.
The ELO gain between each position (bottom to top) is as follows...
303.4, 166.74, 116.35, 106.7, 59.34, 25.74
So clearly decreasing for each doubling in size. ChatGPT extrapolated +12.9 for 128 million and +6.4 for 256 million.

So for a simple network like Chess768 there are still some gains to be had with datasets into the hundreds of millions but approaching saturation where time might be better spent elsewhere. I would guess that networks with more complex architectures can keep giving meaningful gains well into the billions of positions.

[Steve Maughan]

Hi Martin — that's interesting. At what point did the NNUE version beat the HCE version?

— Steve

[MartinBryant]

Hi Martin — that's interesting. At what point did the NNUE version beat the HCE version?

— Steve

Hi Steve.
Well I've actually converted enough of Colossus' old test games to quiet positions for the 128 million network now.
So I reran it and also included the last version with the HCE.

Code: Select all

Rank Name                             Elo        +/-       nElo        +/-      Games      Score       Draw
   1 ColossusChess5-128            223.94       5.53     283.55       5.38      16000      78.4%      19.3%
   2 ColossusChess5-64             211.38       5.45     264.39       5.38      16000      77.1%      19.6%
   3 ColossusChess5-32             183.40       5.30     222.86       5.38      16000      74.2%      19.7%
   4 ColossusChess5-16             128.75       5.03     150.72       5.38      16000      67.7%      20.6%
   5 ColossusChess5-8               24.80       4.93      27.16       5.38      16000      53.6%      16.4%
   6 ColossusChess5-4              -82.65       5.18     -89.26       5.38      16000      38.3%      11.9%
   7 ColossusChess2026b            -98.26       5.40    -103.27       5.38      16000      36.2%       6.5%
   8 ColossusChess5-2             -248.67       6.51    -283.43       5.38      16000      19.3%       5.6%
   9 ColossusChess5-1             -506.54      11.57    -744.78       5.38      16000       5.1%       2.0%
Finished match

So it seems a mere 4 million positions is enough to surpass my HCE ... #embarrasing!

The gaps this time for the networks are...
257.87, 166.02, 107.45, 103.95, 54.65, 27.98, 12.56
A bit of a kink in the curve around the 8-->16M mark but still showing the diminishing returns. Indicative of perhaps only 6 or 7 ELO for another doubling.
I guess that either a) experimenting with more sophisticated architectures or b) re-evaluating all the training positions with the new evaluation function and re-training, would now be more productive than generating more positions for this simple model?

[Steve Maughan]

Also interesting. Am I correct in interpreting this was a 300 Elo difference between the HCE and NNUE versions?

— Steve

[MartinBryant]

Also interesting. Am I correct in interpreting this was a 300 Elo difference between the HCE and NNUE versions?

— Steve

Yes. Indeed.
But this isn't new. ChatGPT states that Jamie Whiting, author of the Bullet training software, added a Chess768 NNUE to v0.5 of his Akimbo engine (to become v0.6) back in September 2023 and it jumped from 3026 to 3335 on CCRL. There may be other examples.

[mclane]

Heavens. Colossus NNUE!

I hope I can update my Colossus 4 version and run the new Colossus on my Amstrad CPC 664 !!

[Steve Maughan]

Heavens. Colossus NNUE!

I hope I can update my Colossus 4 version and run the new Colossus on my Amstrad CPC 664 !!

And don’t forget “White Knight NNUE” for the BBC Micro

— Steve

[mclane]

My zep Tepi chess tournament with dedicated units and Homecomputer software has a white Knight version running on an acorn electron.

Code: Select all

:
    Motor                                                       Punkte  Ch St Cy Me Co Gr Mi Ph Ch Sf Ca Ch De Wh Ga De Ch Gr MS Sc Co Sa Fi Ta My Ph Lo At Ma Ps Sc Sc    S-B
01: CheckCheck 1.75 XT mit 350 cycles                           11,0/13 ··    1  0  0  1  1        1  1     1  1           1        1  1              1                   78,00
02: Steinitz Encore Master Chess 4 mhz Applied Concepts         11,0/13    ··    1  =        0  1  1        1  1           1  =  1        1              1  1             72,00
03: Cyrus Dragon Chess {Brettcomputer}                          10,5/13 0     ·· =  =        1  1  1        1              =     1        1     1     1        1          66,75
04: Mephisto ESB II {Brettcomputer}                             10,0/13 1  0  =  ·· 1        0     1     11          1        =     1                    1     1          70,25
05: Conchess 2 mhz {Brettcomputer}                              10,0/13 1  =  =  0  ·· =     1                 1  =  1        1        1                    1        1    68,25
06: Grünfeld S Morphy Capablanca S Applied Concepts             9,0/13  0           =  ·· 0  1  =        =                    1           =     1     1  1  1        1    45,50
07: Millennium Chess Champion 2.0 {Brettcomputer}               8,5/13  0              1  ··    1  0  =  0  1  0  1     1        1  1                          1          56,75
08: PhilidorChess IBM XT 350 cycles                             8,5/13     1  0  1  0  0     ··       1           =     1        1     0  1                 1        1    55,00
09: Chess Mate Sam Coupe {Brettcomputer}                        8,5/14     0  0        =  0     ··    1  1        1  0  1              =     1  =                 1  1    48,25
10: Sfinks4 TRS-80 {Brettcomputer}                              8,0/13  0  0  0  0        1        ··             1     1  1  =     1  =        1                 1       47,00
11: Capa44 286-12 1510 cycles {Brettcomputer}                   7,5/13  0                 =  0  0     ··    1  1  1  =     0     1     =                    1     1       42,75
12: Chess 7.5 C64 {Brettcomputer}                               7,5/13           00    =  1     0        ··    0  0  1        1        1                 1  1     1       39,00
13: Deep Scan Sam Coupe {Brettcomputer}                         7,5/14  0  0  0           0           0     ·· =     =        1           =     1  1  1  1     1          32,50
14: White Knight MK11 Acorn Electron {Brettcomputer}            7,0/13  0  0        0     1           0  1  =  ··    1           =        0     1                 1  1    39,25
15: Gambiet81 TRS80 {Brettcomputer}                             7,0/13              =     0  =  0  0  0  1        ··       =        1  1                 1  =     1       37,25
16: Destiny Prodigy Mini Applied Concepts {Brettcomputer}       7,0/13           0  0           1     =  0  =  0     ··          0        1              1  1  1     1    31,50
17: Chess 3.48 Oric Atmos o/c {Brettcomputer}                   7,0/13                    0  0  0  0                    ·· 1  0     1  1  1              1  =     =  1    30,00
18: Grandmaster Chess C64 {Brettcomputer}                       6,5/14  0  0  =                    0  1           =     0  ·· 0  1     =           1  1     =        =    34,25
19: MSX Chess V1.1                                              6,0/14     =     =  0  0           =     0  0           1  1  ·· =           =        0        =     1    37,25
20: Scisys President Chess {Brettcomputer}                      6,0/14     0  0           0  0        0        =     1     0  =  ··    1     =  =     1        1          28,75
21: Conic Computer Chess Modell 7012 {Brettcomputer}            5,5/14  0        0        0        0              0     0           ··    =        =  1  =  0  11    1    16,50
22: Sargon 2.5 Applied Concepts / Chafitz {Brettcomputer}       5,0/14  0           0        1  =  =  =  0        0     0  =     0     ·· 1  0  1                         33,25
23: Fidelity Chess Challenger Grandmaster Voice {Brettcomputer} 5,0/14     0  0        =     0              =  1     0  0           =  0  ··    0  =        1     1       25,00
24: Tacticon TI99/4a                                            5,0/7                           0                             =  =     1     ·· 1        1  1             21,00
25: Mychess DOS IBM XT 350 cycles {Brettcomputer}               4,5/14        0        0        =  0        0  0                 =     0  1  0  ··    1  =     0  1       19,25
26: Phoenix Chess TI99/4a                                       4,0/7                                       0              0        =     =        ··          1  1  1    12,75
27: Logichess2.2 Lynx {Brettcomputer}                           3,0/13  0     0        0                    0              0  1  0  0           0     ·· =     1  =  0    11,25
28: Atari Video Chess VCS {Brettcomputer}                       3,0/14     0     0     0                 0  0     0  0  0           =        0  =     =  ··    =  1       10,25
29: Mattel Chess Aquarius {Brettcomputer}                       2,5/14     0        0  0     0        0  0        =  0  =  =        1     0  0              ··       0    15,75
30: Psi Chess Sinclair ZX Spectrum {Brettcomputer}              2,5/14        0  0        0                 0        0        =  0  00          1  0  0  =     ·· =       10,25
31: SciSys Chess Champion MKIV {Brettcomputer}                  2,5/14                          0  0  0  0     0  0     =                 0     0  0  =  0     =  ·· 1     8,75
31: Scisys Senator Chess                                        2,5/14              0  0     0  0              0     0  0  =  0     0              0  1     1     0  ··    8,75

209 Partien von 496 gespielt
Name des Turniers: Zep tepi summer2024

[Steve Maughan]

LOVE IT!!