Syzygy benefit for current SF

[Jouni]

I ran 1000 games with 10+0,1 and 60+0,6 level to see TB gain for latest SF. 5 + 6 piece tablebases and endgames book.

Code: Select all

Score of SFdev TB vs SFdev: 178 - 163 - 659 [0.507]
...      SFdev TB playing White: 109 - 67 - 324  [0.542] 500
...      SFdev TB playing Black: 69 - 96 - 335  [0.473] 500
...      White vs Black: 205 - 136 - 659  [0.534] 1000
Elo difference: 5.2 +/- 12.6, LOS: 79.2 %, DrawRatio: 65.9 %
1000 of 1000 games finished.

Score of SFdev TB vs SFdev: 181 - 164 - 655 [0.508]
...      SFdev TB playing White: 111 - 58 - 331  [0.553] 500
...      SFdev TB playing Black: 70 - 106 - 324  [0.464] 500
...      White vs Black: 217 - 128 - 655  [0.544] 1000
Elo difference: 5.9 +/- 12.6, LOS: 82.0 %, DrawRatio: 65.5 %
1000 of 1000 games finished.

The latest SF with NNUE seems to get less and less benefit? 5 ELO means 1-2 ELO for from starting position! Previous versions have got always +15 ELO.

[Viren]

On what hardware?

[Jouni]

Hardware i5 4Ghz. I repeated 10+0,1 with classical evaluation and quite different result:

Code: Select all

Score of SFdev classic vs SFdev classic TB: 131 - 231 - 638 [0.450]
...      SFdev classic playing White: 79 - 94 - 327  [0.485] 500
...      SFdev classic playing Black: 52 - 137 - 311  [0.415] 500
...      White vs Black: 216 - 146 - 638  [0.535] 1000
Elo difference: -34.9 +/- 12.9, LOS: 0.0 %, DrawRatio: 63.8 %
1000 of 1000 games finished.

NNUE has a lot of endgame knowledge .

[amanjpro]

Hardware i5 4Ghz. I repeated 10+0,1 with classical evaluation and quite different result:

Code: Select all

Score of SFdev classic vs SFdev classic TB: 131 - 231 - 638 [0.450]
...      SFdev classic playing White: 79 - 94 - 327  [0.485] 500
...      SFdev classic playing Black: 52 - 137 - 311  [0.415] 500
...      White vs Black: 216 - 146 - 638  [0.535] 1000
Elo difference: -34.9 +/- 12.9, LOS: 0.0 %, DrawRatio: 63.8 %
1000 of 1000 games finished.

NNUE has a lot of endgame knowledge .

I believe what is interesting is how did you load EGTB? was it on SSD, RAM or what?

[phhnguyen]

IMHO, the result is not clear since the both Elo and error range is not too far from the Fishtest result (https://github.com/glinscott/fishtest/wiki/UsefulData) when your number of games is significantly smaller (1 K vs 80 K).

Should you test with over 80 K games? Notes that Fishtest loaded all tables into RAM.

[Jouni]

I don't have hardware for 80k games. But isn't LOS 100% for classical more than 80% for NNUE? I hope framework does a new test.