Testing Stockfish 11-03-13. 480 Games.

[Tomcass]

Hello, my friend.

I have finished my test with Sugar X, and started testing Komodo 9.4. I will resume my Stockfish Development tests in a couple of days.

Kind regards from Barcelona,

Tom.

... by the way, I know that Komodo 9.4 has some minor bugs, but I will finish my 960 games test anyway.

[Ozymandias]

started testing Komodo 9.4
[…]
…. by the way, I know that Komodo 9.4 has some minor bugs, but I will finish my 960 games test anyway.

If they are crippling the engine by some 10 ELO points, they aren't so minor.

[Tomcass]

You are right, Juan. The Komodo's bugs are more than 'minor'. Let's hope that 9.42 will be OK.

By the way ...

A big THANKS to the Computer Chess Community for the attention given to this thread. More than 200.000 visits are a big reason to keep testing.



Kind regards from Barcelona.

Tom.

[Ozymandias]

Another milestone.

[Tomcass]

TESTING STOCKFISH DEVELOPMENT 240416: : 1920 GAMES

Timestamp: 1461456058 Bench: 7890808

FIRST LEG. FIXED TIME CONTROL. 960 GAMES

6 real cores
Ponder: Off.
GUI: Fritz 14
Book: Perfect 2014c – Limit 5 moves. Sedat –
No tablebases. No RTB used.
Hash 512
Relative Speed: 29.54
Knodes per second: 14.177

Time Control 4+0
Stockfish 240416 64 POPCNT_6 - Komodo 9.42 64-bit_nob_X6 65.5 - 54.5 +28/=75/-17 54.58%
Stockfish 240416 64 POPCNT_6 - SugaR 2.0 64 POPCNT_X6 62.5 - 57.5 +11/=103/-6 52.08%
Stockfish 240416 64 POPCNT_6 - Houdini 4 Pro x64_St_Ct0_X6 82.5 - 37.5 +50/=65/-5 68.75%
Stockfish 240416 64 POPCNT_6 - Gull 3 x64 XP_X6 83.5 - 36.5 +54/=59/-7 69.58%

Score after 480 Games using 6 cores= 294.0 – 186.0 = 61.25%
480 Games= http://www.mediafire.com/download/rticbq6qz35mz2u/

8 real cores
Ponder: Off
GUI: Fritz 14
Book: Perfect 2014c – Limit 5 moves. Sedat –
No tablebases. No RTB used.
Hash 1024
Relative Speed: 53.22
Knodes per Second: 25.543

Fixed Time Control 4+0
Stockfish 240416 64 BMI2_8 - Komodo 9.42 64-bit_X8_nob 65.0 - 55.0 +27/=76/-17 54.17%
Stockfish 240416 64 BMI2_8 - SugaR 2.0 64 BMI2_X8 65.5 - 54.5 +12/=107/-1 54.58%
Stockfish 240416 64 BMI2_8 - Houdini 4 Pro x64_Ct0_8 84.0 - 36.0 +51/=66/-3 70.00%
Stockfish 240416 64 BMI2_8 - Gull 3 x64 BMI2_X8 87.0 - 33.0 +58/=58/-4 72.50%

Score after 480 games using 8 cores: 301.5 – 178.5 = 62.82%
480 Games= http://www.mediafire.com/download/v8vhurdarcddd0g

Global score for Stockfish 240416 after 960 games : 595.5 - 364.5= 62.03%

AVERAGE ELO OF OPONENTS= 3.196

ESTIMATED ELO FOR STOCKFISH DEV 240416 = 3.280

1.- Stockfish Development 020316= 3.287
2.- SugaR 2.0= 3.285
3.- Stockfish Development 150116 IPMAN= 3.283
4.- SugaR Pro v1.3= 3.280
5.- Stockfish 7 = 3.268
6.- Komodo 9.3 = 3.259

Let’s start the second leg at incremental time control.

Regards,

Tom.

[Kohflote]

Hi Tom,

Just wondering what is the result of the 2nd leg test. Thank you.

Best regards,
Koh, Kah Huat

[Tomcass]

Hello, my friend. Here you have the second leg and the global score.

TESTING STOCKFISH DEVELOPMENT 240416: : 1920 GAMES
Timestamp: 1461456058 Bench: 7890808
SECOND LEG. INCREMENTAL TIME CONTROL. 960 GAMES

6 real cores
Ponder: Off.
GUI: Fritz 14
Book: Perfect 2014c – Limit 5 moves. Sedat –
No tablebases. No RTB used.
Hash 512
Relative Speed: 29.54
Knodes per second: 14.177

Time Control 2+2

Stockfish 240416 64 POPCNT_6 - Komodo 9.42 64-bit_nob_X6 70.5 - 49.5 +39/=63/-18 58.75%
Stockfish 240416 64 POPCNT_6 - SugaR 2.0 64 POPCNT_X6 61.0 - 59.0 +8/=106/-6 50.83%
Stockfish 240416 64 POPCNT_6 - Houdini 4 Pro x64_St_Ct0_X6 83.5 - 36.5 +55/=57/-8 69.58%
Stockfish 240416 64 POPCNT_6 - Gull 3 x64 XP_X6 87.5 - 32.5 +57/=61/-2 72.92%

Score after 480 Games using 6 cores= 302.5 – 177.5 = 63.02%
480 Games= http://www.mediafire.com/download/clge06ankak7azs

8 real cores
Ponder: Off
GUI: Fritz 14
Book: Perfect 2014c – Limit 5 moves. Sedat –
No tablebases. No RTB used.
Hash 1024
Relative Speed: 53.22
Knodes per Second: 25.543

Time Control 2+2

Stockfish 310316 64 BMI2_X8 - Komodo 9.42 64-bit_X8_nob 70.5 - 49.5 +28/=85/-7 58.75%
Stockfish 310316 64 BMI2_X8 - SugaR 2.0 64 BMI2_X8 63.0 - 57.0 +9/=108/-3 52.50%
Stockfish 310316 64 BMI2_X8 - Houdini 4 Pro x64_Ct0_8 85.5 - 34.5 +53/=65/-2 71.25%
Stockfish 310316 64 BMI2_X8 - Gull 3 x64 BMI2_X8 87.0 - 33.0 +56/=62/-2 72.50%
Score after 480 Games using 8 cores= 306.0 – 174.0 = 63.75%
480 Games= http://www.mediafire.com/download/jygbd407tlyefzh/

Global score for Stockfish 240416 after 960 games : 608.5 – 351.5= 63.39%
AVERAGE ELO OF OPONENTS= 3.196
ESTIMATED ELO FOR STOCKFISH DEV 240416 = 3.290

------------------------------

FINAL SCORE FOR STOCKFISH 240416 AFTER 1.920 GAMES = 1.204.0 – 716.0= 62.71%
AVERAGE ELO OF OPPONENTS= 3.196
ESTIMATED ELO FOR STOCKFISH DEV 240416 AFTER 1.920 GAMES= 3.185

MY RANKING OF SELECTED ENGINES

1.- Stockfish Development 020316= 3.287
2.- Stockfish Development 240416= 3.285
3.- SugaR 2.0= 3.285
4.- Stockfish Development 150116 IPMAN= 3.283
5.- Stockfish 7 = 3.268
6.- Komodo 9.42 = 3.261
7.- Komodo 9.3 = 3.259
8.- Komodo 9.2 = 3.251.
9.- Komodo 9.1= 3.247
10.- DON 190216 = 3.242
11.- Stockfish 6= 3.219
12.- Komodo 9= 3.209
13.- Houdini 4= 3.136
14.- Gull 3 Development 150216= 3.120
15.- Gull 3 XP= 3.103

Best regards from Barcelona.

Tom.

[Tomcass]

TESTING STOCKFISH IPMAN 050516 : 960 GAMES

6 real cores
Ponder: Off.
GUI: Fritz 14
Book: Perfect 2014c – Limit 5 moves. Sedat –
No tablebases. No RTB used.
Hash 512
Relative Speed: 29.54
Knodes per second: 14.177

Fixed Time Control 4+0

Stockfish 050516IP 64 POPCNT_ - Komodo 9.42 64-bit_nob_X6 34.0 - 26.0 +15/=38/-7 56.67%
Stockfish 050516IP 64 POPCNT_ - SugaR 2.0 64 POPCNT_X6 32.0 - 28.0 +9/=46/-5 53.33%
Stockfish 050516IP 64 POPCNT_ - Houdini 4 Pro x64_St_Ct0_X6 47.0 - 13.0 +35/=24/-1 78.33%
Stockfish 050516IP 64 POPCNT_ - Gull 3 x64 XP_X6 46.5 - 13.5 +34/=25/-1 77.50%

Incremental Time Control 2+2

Stockfish 050516IP 64 POPCNT_ - Komodo 9.42 64-bit_nob_X6 36.0 - 24.0 +15/=42/-3 60.00%
Stockfish 050516IP 64 POPCNT_ - SugaR 2.0 64 POPCNT_X6 33.0 - 27.0 +9/=48/-3 55.00%
Stockfish 050516IP 64 POPCNT_ - Houdini 4 Pro x64_St_Ct0_X6 40.5 - 19.5 +23/=35/-2 67.50%
Stockfish 050516IP 64 POPCNT_ - Gull 3 x64 XP_X6 45.5 - 14.5 +31/=29/-0 75.83%

Score after 480 Games using 6 cores= 314.5 – 165.5= 65.52%
480 Games= http://www.mediafire.com/download/5kdvaay0qp2scim

8 real cores
Ponder: Off
GUI: Fritz 14
Book: Perfect 2014c – Limit 5 moves. Sedat –
No tablebases. No RTB used.
Hash 1024
Relative Speed: 53.22
Knodes per Second: 25.543

Fixed Time Control 4+0

Stockfish 050516IP 64 bmi2_8 - Komodo 9.42 64-bit_X8_nob 40.0 - 20.0 +22/=36/-2 66.67%
Stockfish 050516IP 64 bmi2_8 - SugaR 2.0 64 BMI2_X8 30.0 - 30.0 +3/=54/-3 50.00%
Stockfish 050516IP 64 bmi2_8 - Houdini 4 Pro x64_Ct0_8 43.0 - 17.0 +28/=30/-2 71.67%
Stockfish 050516IP 64 bmi2_8 - Gull 3 x64 BMI2_X8 44.5 - 15.5 +31/=27/-2 74.17%

Incremental Time Control= 2+2

Stockfish 050516IP 64 bmi2_8 - Komodo 9.42 64-bit_X8_nob 32.0 - 28.0 +13/=38/-9 53.33%
Stockfish 050516IP 64 bmi2_8 - SugaR 2.0 64 BMI2_X8 31.0 - 29.0 +4/=54/-2 51.67%
Stockfish 050516IP 64 bmi2_8 - Houdini 4 Pro x64_Ct0_8 42.5 - 17.5 +26/=33/-1 70.83%
Stockfish 050516IP 64 bmi2_8 - Gull 3 x64 BMI2_X8 43.0 - 17.0 +27/=32/-1 71.67%

480 Games= http://www.mediafire.com/download/io4ofjaxqrcrxo2
Score after 480 games using 8 cores: 306-0 – 174.0= 63.75%

SCORE AFTER 960 GAMES = 620.5 – 339.5= 64.64%
AVERAGE ELO OF OPONENTS= 3.196
ESTIMATED ELO FOR STOCKFISH IPMAN 050516= 3.298

This score is impressive. 11 elo poins above the best Stockfish Development in my tests so far, and 7 elo points better than my current leader SugaR 2.6. I will repeat this test to reduce the error bars.

Regards,

Tom.

[beram]

TESTING STOCKFISH IPMAN 050516 : 960 GAMES

6 real cores
Ponder: Off.
GUI: Fritz 14
Book: Perfect 2014c – Limit 5 moves. Sedat –
No tablebases. No RTB used.
Hash 512
Relative Speed: 29.54
Knodes per second: 14.177

Fixed Time Control 4+0

Stockfish 050516IP 64 POPCNT_ - Komodo 9.42 64-bit_nob_X6 34.0 - 26.0 +15/=38/-7 56.67%
Stockfish 050516IP 64 POPCNT_ - SugaR 2.0 64 POPCNT_X6 32.0 - 28.0 +9/=46/-5 53.33%
Stockfish 050516IP 64 POPCNT_ - Houdini 4 Pro x64_St_Ct0_X6 47.0 - 13.0 +35/=24/-1 78.33%
Stockfish 050516IP 64 POPCNT_ - Gull 3 x64 XP_X6 46.5 - 13.5 +34/=25/-1 77.50%

Incremental Time Control 2+2

Stockfish 050516IP 64 POPCNT_ - Komodo 9.42 64-bit_nob_X6 36.0 - 24.0 +15/=42/-3 60.00%
Stockfish 050516IP 64 POPCNT_ - SugaR 2.0 64 POPCNT_X6 33.0 - 27.0 +9/=48/-3 55.00%
Stockfish 050516IP 64 POPCNT_ - Houdini 4 Pro x64_St_Ct0_X6 40.5 - 19.5 +23/=35/-2 67.50%
Stockfish 050516IP 64 POPCNT_ - Gull 3 x64 XP_X6 45.5 - 14.5 +31/=29/-0 75.83%

Score after 480 Games using 6 cores= 314.5 – 165.5= 65.52%
480 Games= http://www.mediafire.com/download/5kdvaay0qp2scim

8 real cores
Ponder: Off
GUI: Fritz 14
Book: Perfect 2014c – Limit 5 moves. Sedat –
No tablebases. No RTB used.
Hash 1024
Relative Speed: 53.22
Knodes per Second: 25.543

Fixed Time Control 4+0

Stockfish 050516IP 64 bmi2_8 - Komodo 9.42 64-bit_X8_nob 40.0 - 20.0 +22/=36/-2 66.67%
Stockfish 050516IP 64 bmi2_8 - SugaR 2.0 64 BMI2_X8 30.0 - 30.0 +3/=54/-3 50.00%
Stockfish 050516IP 64 bmi2_8 - Houdini 4 Pro x64_Ct0_8 43.0 - 17.0 +28/=30/-2 71.67%
Stockfish 050516IP 64 bmi2_8 - Gull 3 x64 BMI2_X8 44.5 - 15.5 +31/=27/-2 74.17%

Incremental Time Control= 2+2

Stockfish 050516IP 64 bmi2_8 - Komodo 9.42 64-bit_X8_nob 32.0 - 28.0 +13/=38/-9 53.33%
Stockfish 050516IP 64 bmi2_8 - SugaR 2.0 64 BMI2_X8 31.0 - 29.0 +4/=54/-2 51.67%
Stockfish 050516IP 64 bmi2_8 - Houdini 4 Pro x64_Ct0_8 42.5 - 17.5 +26/=33/-1 70.83%
Stockfish 050516IP 64 bmi2_8 - Gull 3 x64 BMI2_X8 43.0 - 17.0 +27/=32/-1 71.67%

480 Games= http://www.mediafire.com/download/io4ofjaxqrcrxo2
Score after 480 games using 8 cores: 306-0 – 174.0= 63.75%

SCORE AFTER 960 GAMES = 620.5 – 339.5= 64.64%
AVERAGE ELO OF OPONENTS= 3.196
ESTIMATED ELO FOR STOCKFISH IPMAN 050516= 3.298

This score is impressive. 11 elo poins above the best Stockfish Development in my tests so far, and 7 elo points better than my current leader SugaR 2.6. I will repeat this test to reduce the error bars.

Regards,

Tom.

Nice work Tom
I have done some research in your ongoing thread
And have constructed a graph visual for the difference between the mean opponents in your list based on Stockfish and Komodo

http://postimg.org/image/yi4m1tbr5/

[Tomcass]

Thanks for this excellent Visual Graph, Bram!. What a useful tool to see the evolution of both programs.

Kind regards from Barcelona.

Tom.