Fun fact: SF NN net beats syzygy files easily!

[Jouni]

I had feeling, that SF NNUE net is for openings and middle games. But NO. Simple test shows, that SF NNUE plays endgames about 40 ELO better than classic! So with 20 MB You got more ELO gain than with giga/terabytes of syzygy tables.

[h1a8]

Can you please post results of these tests including the test sets themselves?

[syzygy]

And how much for NNUE+TBs vs NNUE?

[Jouni]

Just one result with 12.8. Stockfish. Endgames.epd from framework and 60+0.6 games:

Code: Select all

  

                      
1   SF NNUE      +42  +49/=126/-25 56.00%  112.0/200
2   SF Classic   -42  +25/=126/-49 44.00%   88.0/200

I got same type results in 4 other matches. I also gave Classic 5 piece syzygy and it improved about 20 ELO losing still to NNUE.

[Nay Lin Tun]

OP, what do you believe now? Stockfish is carrying 20MB of endgame tablebase or 20 MB of opening book?

( I know you are the guy who believe Leela is carrying 150 MB of opening book).

[phhnguyen]

Interesting test and info!

However, I am a bit doubtful. On one hand Syzygy 5 men may not contribute much, only 2 Elo as the Fishtest revealed. Of course, Fishtest tested based on the tournament full games, different here, we tested on endgames only but the one having Syzygy did have many benefits anyway. On the other hand, endgame testing depends much on setting up, say, games per endgame, timers, on/off GUIs adjudication... When the materials become unbalanced, which happens high frequently in endgames, SF may auto switch to classical evaluation, two versions become identical, the test becomes using vs not using Syzygy. All require us to test more carefully.

[phhnguyen]

I have just done a very quick test and it confirms your test result: NNUE without Syzygy vs SF clasic with Syzygy 5-men, 200 games 40/20+1, using Endgames.epd from SF repo as openings, each FEN was played twice with side changed. The result as below:

Code: Select all

  #  name                      games    wins   draws  losses  score%    los%  elo+/-
  1. Stockfish NNUE no Syz       200      40     132      28    53.0    92.7    20.9
  2. Stockfish Classic w Syzygy  200      28     132      40    47.0     7.3   -20.9

[phhnguyen]

I have checked how NNUE and Syzygy work. I used two engines to analyze some positions from Endgames.epd. Below is the screen. Stockfish Classic has over 76 K tbhits, Stockfish NNUE has zero tbhits but about 75.0% (750) NNUE evaluation hits. Look like both engines work well as we want.

[Jouni]

For syzygy:

Code: Select all

                       
1   SF NNUE TB   +21  +222/=615/-163 52.95%  529.5/1000
2   SF NNUE      -21  +163/=615/-222 47.05%  470.5/1000

So TBs still help also NNUE.

[syzygy]

For syzygy:

Code: Select all

                       
1   SF NNUE TB   +21  +222/=615/-163 52.95%  529.5/1000
2   SF NNUE      -21  +163/=615/-222 47.05%  470.5/1000

So TBs still help also NNUE.

Thanks for running the test!