Stockfish NNUE SV Tests

[mehmet123]

The performance of Stockfish NNUE SV Exp 20200722-1130 net against Stockfish 170720 x64 bmi2

Total 1025 games with Balsa 5 move opening book
3 different time controls ( 30 sec +0. 5 sec, 1 min + 0.5 sec, 2 min + 0.5 sec)
Average elo difference is ~58 elo

For now this is a record for my tests.

[mehmet123]

Test of SV 20200724-0123 Net

Program Elo + - Games Score Av.Op. Draws

1 Stockfish NNUE 190720 bmi2 : 2430 16 16 600 58.6 % 2370 64.8 %
2 Stockfish 17072020 x64 bmi2 : 2370 16 16 600 41.4 % 2430 64.8 %

Individual statistics:

1 Stockfish NNUE 190720 bmi2: 2430 600 (+157,=389,- 54), 58.6 %

Stockfish 17072020 x64 bmi2 : 600 (+157,=389,- 54), 58.6 %

2 Stockfish 17072020 x64 bmi2 : 2370 600 (+ 54,=389,-157), 41.4 %

Stockfish NNUE 190720 bmi2 : 600 (+ 54,=389,-157), 41.4 %


Cutechess Gui, 1 Core (i7 9750h), 30 sec + 0.5 sec TC, Balsa 5 Move Opening Book, 64 Mb Hash, Ponder Off

http://www.mediafire.com/file/dfgw4zw5j ... 1.pgn/file

[carldaman]

The performance of Stockfish NNUE SV Exp 20200722-1130 net against Stockfish 170720 x64 bmi2

Total 1025 games with Balsa 5 move opening book
3 different time controls ( 30 sec +0. 5 sec, 1 min + 0.5 sec, 2 min + 0.5 sec)
Average elo difference is ~58 elo

For now this is a record for my tests.

So, are you saying this experimental net outperformed the regular SV nnue nets?

[mehmet123]

Yes, for now it's the strongest Stockfish NNUE net according to my tests.
But evertyhing can change very fast. Because new SV nets are published at regular time intervals.

[carldaman]

Interesting that the experimental net is the best, at least for now. Thanks for doing this testing - especially with possible regressions lurking, it's hard to know if the latest is also the best.

[mehmet123]

Test of SV 20200724-2344 Net

Program Elo + - Games Score Av.Op. Draws

1 Stockfish NNUE 190720 bmi2 : 2442 24 24 360 61.9 % 2358 55.6 %
2 Stockfish 17072020 x64 bmi2 : 2358 24 24 360 38.1 % 2442 55.6 %


Individual statistics:

1 Stockfish NNUE 190720 bmi2: 2442 360 (+123,=200,- 37), 61.9 %

Stockfish 17072020 x64 bmi2 : 360 (+123,=200,- 37), 61.9 %

2 Stockfish 17072020 x64 bmi2 : 2358 360 (+ 37,=200,-123), 38.1 %

Stockfish NNUE 190720 bmi2 : 360 (+ 37,=200,-123), 38.1 %


Cutechess Gui, 1 Core (i7 9750h), 30 sec + 0.5 sec TC, Balsa 5 Move Opening Book, 16 Mb Hash, Ponder Off

http://www.mediafire.com/file/72p94020j ... 0.pgn/file


Very great result of Stockfish NNUE (20200724-2344). 84 elo difference is a new record for my tests.

[mehmet123]

Don't forget this these tests has been made with 1 (core i7 9750h) with short time controls (30 sec + 0.5 sec , 1 min + 0.5 sec, 2 min + 0.5).
At long time control matches we shouldn't expect such a big difference elo.
In my tests the average elo performance of SF NNUE Gekehenker is +32 elo against Stockfish Dev. I had made some little test with SF NNUE Gekehenker at 10 minutes time control. I found a difference of + 17 elo performance against Stockfish Dev.
SF NNUE is getting stronger, elo difference will gradually increase in long-term matches.

[mehmet123]

Test of SV 20200724-2344 Net

Program Elo + - Games Score Av.Op. Draws

1 Stockfish NNUE SV2344 : 2424 28 26 150 57.0 % 2376 75.3 %
2 Stockfish 170720 x64 bmi2 : 2376 26 28 150 43.0 % 2424 75.3 %

Individual statistics:

1 Stockfish NNUE SV2344 : 2424 150 (+ 29,=113,- 8), 57.0 %

Stockfish 170720 x64 bmi2 : 150 (+ 29,=113,- 8), 57.0 %

2 Stockfish 170720 x64 bmi2 : 2376 150 (+ 8,=113,- 29), 43.0 %

Stockfish NNUE SV2344 : 150 (+ 8,=113,- 29), 43.0 %


Arena Gui, 6 Core (i7 9750h), 2 min + 1 sec TC, Balsa 5 Move Opening Book,512 Mb Hash, Ponder Off

http://www.mediafire.com/file/eoh2keixc ... 0.pgn/file


I tested Stockfish NNUE SV2344 with 6 cores (2min + 1 sec). 48 elo difference is very good for these conditions

[mehmet123]

Test of SV 20200723-0511 Net

Program Elo + - Games Score Av.Op. Draws

1 Stockfish NNUE SV 0511 : 2439 20 19 440 60.9 % 2361 62.7 %
2 Stockfish 170720 x64 bmi2 : 2361 19 20 440 39.1 % 2439 62.7 %

Individual statistics:

1 Stockfish NNUE SV 0511 : 2439 440 (+130,=276,- 34), 60.9 %

Stockfish 170720 x64 bmi2 : 440 (+130,=276,- 34), 60.9 %

2 Stockfish 170720 x64 bmi2 : 2361 440 (+ 34,=276,-130), 39.1 %

Stockfish NNUE SV 0511 : 440 (+ 34,=276,-130), 39.1 %


Cutechess Gui, 1 Core (i7 9750h), 30 sec + 0.5 sec TC, Balsa 5 Move Opening Book, 64 Mb Hash, Ponder Off

http://www.mediafire.com/file/utu61s9o8 ... 1.pgn/file


T

[mehmet123]

Test of SV 20200723-0511 Net

Program Elo + - Games Score Av.Op. Draws

1 Stockfish NNUE SV 0511 : 2426 26 24 150 57.3 % 2374 78.7 %
2 Stockfish 170720 x64 bmi2 : 2374 24 26 150 42.7 % 2426 78.7 %

Individual statistics:

1 Stockfish NNUE SV 0511 : 2426 150 (+ 27,=118,- 5), 57.3 %

Stockfish 170720 x64 bmi2 : 150 (+ 27,=118,- 5), 57.3 %

2 Stockfish 170720 x64 bmi2 : 2374 150 (+ 5,=118,- 27), 42.7 %

Stockfish NNUE SV 0511 : 150 (+ 5,=118,- 27), 42.7 %


Arena Gui, 6 Core (i7 9750h), 2 min + 1 sec TC, Balsa 5 Move Opening Book,512 Mb Hash, Ponder Off
http://www.mediafire.com/file/e07i9880k ... 1.pgn/file

52 elo difference is a new record at these conditions (6 core , 2 min + 1 sec).