Stockfish NNUE SV Tests

[carldaman]

Game Conditions: Cutechess Gui, 1 Core (Core-i7 9750h), 2 min + 0.5 sec TC, Balsa 5 Moves Opening Book, 128 Mb Hash, Ponder Off//Default Net
Cfish version:Cfish 131220 x64 E BMI2 mingw10 (ChessMan version)
https://www.mediafire.com/file/phz4bvdd ... 5.pgn/file

This is a different experience file than Exp3 file. They are basically close files but there are some differences in game database and settings. Minimum experience depth is 16 for this file. For Exp3 file the game database is smaller than V16 file and minimum experience depth is 18.
Too many games in experience file does not mean the file will be stronger. Sometimes ineffective games weaken the experience file.

Interesting observations, but what do you mean by ''ineffective games" ?

Shouldn't the learning process weed out the bad moves or lines, anyway?
If such games weaken the exp file, then it would be a sign of ineffective learning, would it not?

But don't get me wrong, I very much like and support all these engines and authors who have implemented learning.
I hope they succeed in their mission.

[mehmet123]

Interesting observations, but what do you mean by ''ineffective games" ?

Shouldn't the learning process weed out the bad moves or lines, anyway?
If such games weaken the exp file, then it would be a sign of ineffective learning, would it not?

But don't get me wrong, I very much like and support all these engines and authors who have implemented learning.
I hope they succeed in their mission.

This isn't about same experience file. I had prepared an experience file and I have testing this file by merging some other experience files. Some of the experience files don't contribute to a good performance after this merge for my test conditions. Of course, each experience file with contains sufficient number of games will contribute more or less to the chess engine.

[carldaman]

OK, thanks for the clarification.

[mehmet123]

Sugar AI vs Cfish:

Program Elo + - Games Score Av.Op. Draws

1 Cfish 171220 x64 bmi2 : 2404 11 10 300 51.2 % 2396 92.3 %
2 SugaR AI 1.20 bmi2 : 2396 10 11 300 48.8 % 2404 92.3 %


Individual statistics:

1 Cfish 171220 x64 bmi2 : 2404 300 (+ 15,=277,- 8), 51.2 %

SugaR AI 1.20 bmi2 : 300 (+ 15,=277,- 8), 51.2 %

2 SugaR AI 1.20 bmi2 : 2396 300 (+ 8,=277,- 15), 48.8 %

Cfish 171220 x64 bmi2 : 300 (+ 8,=277,- 15), 48.8 %


Game Conditions: Cutechess Gui, 1 Core (Core-i7 9750h), 1 min + 0.5 sec TC, Balsa 5 Moves Opening Book, 128 Mb Hash, Ponder Off
Cfish compile:Cfish 171220 x64 E BMI2 mingw10 (Chessman compile) // Sugar AI wasn't use experince file//Default Nets
https://www.mediafire.com/file/it8qr4uy ... 2.pgn/file

[mehmet123]

SugaR AI v Cfish:

Program Elo + - Games Score Av.Op. Draws

1 SugaR AI 1.00 x64 bmi2 TX1 : 2401 7 7 420 50.2 % 2399 95.2 %
2 Cfish 131220 x64 bmi2 : 2399 7 7 420 49.8 % 2401 95.2 %


Individual statistics:

1 SugaR AI 1.00 x64 bmi2 TX1: 2401 420 (+ 11,=400,- 9), 50.2 %

Cfish 131220 x64 bmi2 : 420 (+ 11,=400,- 9), 50.2 %

2 Cfish 131220 x64 bmi2 : 2399 420 (+ 9,=400,- 11), 49.8 %

SugaR AI 1.00 x64 bmi2 TX1 : 420 (+ 9,=400,- 11), 49.8 %


Game Conditions: Cutechess Gui, 1 Core (Core-i7 9750h), 5 min TC, Balsa 5 Moves Opening Book, 128 Mb Hash, Ponder Off
Cfish compile:Cfish 131220 x64 E BMI2 mingw10 (Chessman compile) // Sugar AI played with TX1 exp file//Default Nets
https://www.mediafire.com/file/b53zrwc6 ... 1.pgn/file

The experience files I used in my previous tests were very large compared to these files. They have a value between 100 MB and 1 GB but this experience file (TX1) is very small compared to other files, although it is only 8.5 MB. Sugar AI with this exp file managed to perform well in the match under these conditions( 5 minutes/game)

[mehmet123]

SugaR AI vs Stockfish (Fischer Random Chess Match)

Program Elo + - Games Score Av.Op. Draws

1 SugaR AI 1.20 x64 bmi2 : 2403 12 12 1000 51.0 % 2397 69.0 %
2 Stockfish 251220 x64 bmi2 : 2397 12 12 1000 49.0 % 2403 69.0 %

Individual statistics:

1 SugaR AI 1.20 x64 bmi2 : 2403 1000 (+165,=690,-145), 51.0 %

Stockfish 251220 x64 bmi2 : 1000 (+165,=690,-145), 51.0 %

2 Stockfish 251220 x64 bmi2 : 2397 1000 (+145,=690,-165), 49.0 %

SugaR AI 1.20 x64 bmi2 : 1000 (+145,=690,-165), 49.0 %


Game Conditions: Cutechess Gui, 1 Core ( i7 9750h), 10 sec + 0.2 sec TC, Chess960 Book 3 Moves, 64 Mb Hash, Ponder Off
Default Nets// SugaR AI played without experience file
https://www.mediafire.com/file/40wvzfv8 ... 8.pgn/file

[mehmet123]

SugaR AI vs Stockfish:

Program Elo + - Games Score Av.Op. Draws

1 SugaR AI 1.30 x64 bmi2 : 2401 7 7 1000 50.1 % 2399 88.7 %
2 Stockfish 311220 x64 bmi2 : 2399 7 7 1000 49.9 % 2401 88.7 %

Individual statistics:

1 SugaR AI 1.30 x64 bmi2 : 2401 1000 (+ 58,=887,- 55), 50.1 %

Stockfish 311220 x64 bmi2 : 1000 (+ 58,=887,- 55), 50.1 %

2 Stockfish 311220 x64 bmi2 : 2399 1000 (+ 55,=887,- 58), 49.9 %

SugaR AI 1.30 x64 bmi2 : 1000 (+ 55,=887,- 58), 49.9 %


Game Conditions: Cutechess Gui, 1 Core (Core-i7 9750h), 30 sec + 0.5 sec TC, Balsa 5 Moves Opening Book, 64 Mb Hash, Ponder Off
Sugar AI didn' t use experince file//Default Nets
https://www.mediafire.com/file/zmdz0u6t ... 1.pgn/file

[mehmet123]

Cfish vs Sugar AI:

Program Elo + - Games Score Av.Op. Draws

1 Cfish 040121 x64 bmi2 : 2401 4 4 950 50.2 % 2399 96.8 %
2 SugaR AI 1.40 bmi2 : 2399 4 4 950 49.8 % 2401 96.8 %

Individual statistics:

1 Cfish 040121 x64 bmi2 : 2401 950 (+ 17,=920,- 13), 50.2 %

SugaR AI 1.40 bmi2 : 950 (+ 17,=920,- 13), 50.2 %

2 SugaR AI 1.40 bmi2 : 2399 950 (+ 13,=920,- 17), 49.8 %

Cfish 040121 x64 bmi2 : 950 (+ 13,=920,- 17), 49.8 %


Game Conditions: Cutechess Gui, 1 Core (Core-i7 9750h), 5 min TC, Balsa 5 Moves Opening Book, 512 Mb Hash, Ponder Off
Cfish Version: Cfish Ext 040121 x64 ELTO BMI2 (ChessMan version)
Sugar AI 1.40 didn' t use experince file//Default Nets
https://www.mediafire.com/file/agnwvsqp ... 2.pgn/file

This is the best performance of any chess engine against Cfish in my tests. Sugar AI 1.00 had beaten Cfish 131220 in my previous test, but at that time Sugar AI was using learning file.

[mehmet123]

Eman vs Stockfish:

Program Elo + - Games Score Av.Op. Draws

1 Eman 6.90 x64 bmi2 : 2403 7 7 1000 50.7 % 2397 90.5 %
2 Stockfish 300121 x64 bmi2 : 2397 7 7 1000 49.2 % 2403 90.5 %

Individual statistics:

1 Eman 6.90 x64 bmi2 : 2403 1000 (+ 55,=905,- 40), 50.8 %

Stockfish 300121 x64 bmi2 : 1000 (+ 55,=905,- 40), 50.7 %

2 Stockfish 300121 x64 bmi2 : 2397 1000 (+ 40,=905,- 55), 49.2 %

Eman 6.90 x64 bmi2 : 1000 (+ 40,=905,- 55), 49.2 %


Game Conditions: Cutechess Gui, 1 Core (Core-i7 9750h), 30 sec + 0.5 sec TC, Balsa 5 Moves Opening Book, 128 Mb Hash, Ponder Off
Eman 6.90 didn' t use experince file//Default Nets
https://www.mediafire.com/file/dirh5l7h ... 3.pgn/file

[mehmet123]

15 years change of chess engines :

Program Elo + - Games Score Av.Op. Draws

1 Stockfish 310121 x64 bmi2 : 1040 0 0 200 99.8 % 2100 0.5 %
2 Rybka 1.0 Beta x64 : 0 0 0 200 0.2 % 2700 0.5 %

Individual statistics:

1 Stockfish 310121 x64 bmi2 : 1040 200 (+199,= 1,- 0), 99.8 %

Rybka 1.0 Beta x64 : 200 (+199,= 1,- 0), 99.8 %

2 Rybka 1.0 Beta x64 : 0 200 (+ 0,= 1,-199), 0.2 %

Stockfish 310121 x64 bmi2 : 200 (+ 0,= 1,-199), 0.2 %


Game Conditions: Cutechess Gui, 1 Core ( i7 9750h), 1 min + 0.5 sec TC, Balsa 5 Move Opening Book, 128 Mb Hash, Ponder Off
Stockfish 310121 x64 bmi2 played with default nnue and contempt:60 setting
https://www.mediafire.com/file/a5ndxd30 ... 4.pgn/file

Rybka 1.0 x64 was the most powerful chess engine as of February 2006. Rybka 1.0 Beta x64 is +53 elo stronger than Fritz 10 at 1 cpu (Cegt 40/20). Deep Fritz 10 defeated world chess champion Kramnik in 2005 with a score of 4-2.