Testing Stockfish 11-03-13. 480 Games.

Discussion of computer chess matches and engine tournaments.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
JJJ
Posts: 1316
Joined: Sat Apr 19, 2014 11:47 am

Re: For Tom- or Maybe Anyone Who Knows!!

Post by JJJ » Sun May 15, 2016 6:38 am

The total difference is stable and reduced since a long time now.
The next Komodo might be released soon I heard.

Tomcass
Posts: 786
Joined: Sun Apr 16, 2006 7:09 pm

Re: For Tom- or Maybe Anyone Who Knows!!

Post by Tomcass » Fri May 20, 2016 4:30 pm

TESTING STOCKFISH IPMAN 050516 : 1920 GAMES
SECOND LEG: 960 GAMES


6 real cores
Ponder: Off.
GUI: Fritz 14
Book: Perfect 2014c – Limit 5 moves. Sedat –
No tablebases. No RTB used.
Hash 512
Relative Speed: 29.54
Knodes per second: 14.177

Time Control 4+0
Stockfish 050516IP 64 POPCNT_ - Komodo 9.42 64-bit_nob_X6 37.0 - 23.0 +19/=36/-5 61.67%
Stockfish 050516IP 64 POPCNT_ - SugaR 2.0 64 POPCNT_X6 31.0 - 29.0 +6/=50/-4 51.67%
Stockfish 050516IP 64 POPCNT_ - Houdini 4 Pro x64_St_Ct0_X6 42.5 - 17.5 +28/=29/-3 70.83%
Stockfish 050516IP 64 POPCNT_ - Gull 3 x64 XP_X6 47.0 - 13.0 +34/=26/-0 78.33%

240 Games= http://www.mediafire.com/download/ba1z3ltpncf2f6o

Time Control 2+2
Stockfish 050516IP 64 POPCNT_ - Komodo 9.42 64-bit_nob_X6 38.0 - 22.0 +18/=40/-2 63.33%
Stockfish 050516IP 64 POPCNT_ - SugaR 2.0 64 POPCNT_X6 31.0 - 29.0 +6/=50/-4 51.67%
Stockfish 050516IP 64 POPCNT_ - Houdini 4 Pro x64_St_Ct0_X6 43.0 - 17.0 +28/=30/-2 71.67%
Stockfish 050516IP 64 POPCNT_ - Gull 3 x64 XP_X6 46.0 - 14.0 +32/=28/-0 76.67%

240 Games= http://www.mediafire.com/download/uwea1cilkudpv45
Score after 480 Games using 6 cores= 315.5 – 164.5= 65.73%

8 real cores
Ponder: Off
GUI: Fritz 14
Book: Perfect 2014c – Limit 5 moves. Sedat –
No tablebases. No RTB used.
Hash 1024
Relative Speed: 53.22
Knodes per Second: 25.543

Fixed Time Control 4+0

Stockfish 050516IP 64 bmi2_8 - Komodo 9.4 64-bit_X8_nob 39.0 - 21.0 +22/=34/-4 65.00%
Stockfish 050516IP 64 bmi2_8 - SugaR 2.0 64 BMI2_X8 32.0 - 28.0 +8/=48/-4 53.33%
Stockfish 050516IP 64 bmi2_8 - Houdini 4 Pro x64_Ct0_8 48.5 - 11.5 +38/=21/-1 80.83%
Stockfish 050516IP 64 bmi2_8 - Gull 3 x64 BMI2_X8 45.5 - 14.5 +31/=29/-0 75.83%

Incremental Time Control= 2+2

Stockfish 050516IP 64 bmi2_8 - Komodo 9.42 64-bit_X8_nob 36.5 - 23.5 +16/=41/-3 60.83%
Stockfish 050516IP 64 bmi2_8 - SugaR 2.0 64 BMI2_X8 31.5 - 28.5 +6/=51/-3 52.50%
Stockfish 050516IP 64 bmi2_8 - Houdini 4 Pro x64_Ct0_8 45.5 - 14.5 +31/=29/-0 75.83%
Stockfish 050516IP 64 bmi2_8 - Gull 3 x64 BMI2_X8 44.5 - 15.5 +29/=31/-0 74.17%

Score after 480 games using 8 cores: 323.0 – 157.0 = 67.29%
480 Games= http://www.mediafire.com/download/4wst02nw41k91eh

SCORE AFTER 960 GAMES (SECOND LEG) = 638.5 – 321.5= 66.51%
AVERAGE ELO OF OPONENTS= 3.196
ESTIMATED ELO FOR STOCKFISH DEV 050516 IPMAN= 3.312

-------------------

GLOBAL SCORE FOR STOCKFISH IPMAN 050516 AFTER 1.920 GAMES= 1259.0 – 661.0= 65.57%
AVERAGE ELO OF OPONENTS= 3.196
ESTIMATED ELO FOR STOCKFISH DEV 050516 IPMAN= 3.305


The second leg not only has confirmed the excellent score by SF IPMAN 050516 in the first leg of this test, but it has performed even better. If you give a look to my ranking of selected engines, you will see that this compile by Ipman has a substantial advantage (14 elo points) over the previous leader Sugar 2.6. This is probably the most amazing score I have got after 15 years of testing. Great work, Stockfish Team, and what an impressive compile, Ipman. Well done!.

1.- Stockfish Ipman 050516= 3.305
2.- Sugar 2.6 = 3.291
3.- Stockfish Development 020316= 3.287
4.- Stockfish Development 240416= 3.285
5.- SugaR 2.0= 3.285
6.- Stockfish Development 150116 IPMAN= 3.283
7.- Stockfish 7 = 3.268
8.- Komodo 9.42 = 3.261
9.- Komodo 9.3 = 3.259
10.- Komodo 9.2 = 3.251.
11.- Komodo 9.1= 3.247
12.- DON 190216 = 3.242
13.- Stockfish 6= 3.219
14.- Komodo 9= 3.209
15.- Houdini 4= 3.136
16.- Gull 3 Development 150216= 3.120
17.- Gull 3 XP= 3.103

Regards,

Tom.

Tomcass
Posts: 786
Joined: Sun Apr 16, 2006 7:09 pm

Re: For Tom- or Maybe Anyone Who Knows!!

Post by Tomcass » Tue Nov 01, 2016 9:10 am

After 5 moths very busy at professional level, I am pleased to resume my testing activity. Nice to meet you again! :wink:

TESTING SF DEVELOPMENT 0081016 960 GAMES

6 real cores
Ponder: Off.
GUI: Fritz 14
Book: Perfect 2014c – Limit 5 moves. Sedat –
No tablebases. No RTB used.
Hash 512
Relative Speed: 29.54
Knodes per second: 14.177

Fixed Time Control 4+0

Stockfish 081016 64 POPCNT_6 - Komodo 10.1 64-bit_X6_nob 36.0 - 24.0 +14/=44/-2 60.00%
Stockfish 081016 64 POPCNT_6 - SugaR 2.6 64 POPCNT_X6 31.5 - 28.5 +7/=49/-4 52.50%
Stockfish 081016 64 POPCNT_6 - Houdini 4 Pro x64_St_Ct0_X6 48.0 - 12.0 +37/=22/-1 80.00%
Stockfish 081016 64 POPCNT_6 - Gull 3 x64 XP_X6 51.0 - 9.0 +42/=18/-0 85.00%

Incremental Time Control 2+2

Stockfish 081016 64 POPCNT_6 - Komodo 10.1 64-bit_X6_nob 38.5 - 21.5 +22/=33/-5 64.17%
Stockfish 081016 64 POPCNT_6 - SugaR 2.6 64 POPCNT_X6 34.0 - 26.0 +9/=50/-1 56.67%
Stockfish 081016 64 POPCNT_6 - Houdini 4 Pro x64_St_Ct0_X6 47.5 - 12.5 +36/=23/-1 79.17%
Stockfish 081016 64 POPCNT_6 - Gull 3 x64 XP_X6 44.0 - 16.0 +29/=30/-1 73.33%

480 Games= http://www.mediafire.com/file/qcdocw...81016_480g.pgn
Score using 6 cores: 330.5 – 149.5 = 68.85%

8 real cores
Ponder: Off
GUI: Fritz 14
Book: Perfect 2014c – Limit 5 moves. Sedat –
No tablebases. No RTB used.
Hash 1024
Relative Speed: 53.22
Knodes per Second: 25.543

Fixed Time Control 4+0

Stockfish 081016 64 BMI2_8 - Komodo 10.1 64-bit_X8_NOB 35.0 - 25.0 +16/=38/-6 58.33%
Stockfish 081016 64 BMI2_8 - SugaR 2.6 64 BMI2_X8 32.0 - 28.0 +4/=56/-0 53.33%
Stockfish 081016 64 BMI2_8 - Houdini 4 Pro x64_Ct0_8 51.5 - 8.5 +43/=17/-0 85.83%
Stockfish 081016 64 BMI2_8 - Gull 3 x64 BMI2_X8 47.5 - 12.5 +35/=25/-0 79.17%

Incremental Time Control 2+2

Stockfish 081016 64 BMI2_8 - Komodo 10.1 64-bit_X8_NOB 36.5 - 23.5 +18/=37/-5 60.83%
Stockfish 081016 64 BMI2_8 - SugaR 2.6 64 BMI2_X8 31.0 - 29.0 +5/=52/-3 51.67%
Stockfish 081016 64 BMI2_8 - Houdini 4 Pro x64_Ct0_8 49.0 - 11.0 +41/=16/-3 81.67%
Stockfish 081016 64 BMI2_8 - Gull 3 x64 BMI2_X8 48.0 - 12.0 +36/=24/-0 80.00%

480 games: http://www.mediafire.com/file/5dh2qh3rp6aq4me/
Score using 8 cores: 330.5 – 149.5= 68.85%

Against :
Komodo 10.1 (3.278) = 60.83%
SugaR 2.6 (3.291) = 53.54%
Houdini 4.0 Pro (3.136) = 81.67%
Gull 3 XP (3.103) = 79.38%

Average score of oponents= 3.202

Global score after 960 games: 661.0 - 299.0 or 68.85%

Estimated Elo for Stockfish 081016 after 960 games= 3.334

My latest SF favourite before pausing my testing activity was Stockfish Ipman 050516. It reached a score of 3.305 Elo in my tests.
Therefore, according to my tests, in roughly 5 months Stockfish has improved its performance around 29 elo points. Congratulations to the Stockfish Team!.

Regards,

Tom.

Kohflote
Posts: 205
Joined: Wed Sep 19, 2007 9:07 am
Location: Singapore

Re: For Tom- or Maybe Anyone Who Knows!!

Post by Kohflote » Tue Nov 01, 2016 11:10 pm

Dear Tom,

Welcome back! And I missed your excellent testing.

Will you be testing with the newer version of SF or the recently released SF8 soon?

Thank you & regards,
Koh, Kah Huat

Vinvin
Posts: 4754
Joined: Thu Mar 09, 2006 8:40 am
Full name: Vincent Lejeune

Re: For Tom- or Maybe Anyone Who Knows!!

Post by Vinvin » Wed Nov 02, 2016 12:06 am

Kohflote wrote:Dear Tom,

Welcome back! And I missed your excellent testing.
+1 8-)

Tomcass
Posts: 786
Joined: Sun Apr 16, 2006 7:09 pm

Re: For Tom- or Maybe Anyone Who Knows!!

Post by Tomcass » Mon Nov 21, 2016 6:18 pm

Thank you very much, my friends! :D

Kind regards,

Tom.

Tomcass
Posts: 786
Joined: Sun Apr 16, 2006 7:09 pm

Re: For Tom- or Maybe Anyone Who Knows!!

Post by Tomcass » Mon Nov 21, 2016 6:37 pm

TESTING STOCKFISH 8: 960 GAMES

6 real cores
Ponder: Off.
GUI: Fritz 14
Book: Perfect 2014c – Limit 5 moves. Sedat –
No tablebases. No RTB used.
Hash 512
Relative Speed: 29.54
Knodes per second: 14.177

Fixed Time Control 4+0
Stockfish 8 64 POPCNT_X6 - Komodo 10.2 64-bit_X6_NOB 37.5 - 22.5 +18/=39/-3 62.50%
Stockfish 8 64 POPCNT_X6 - SugaR 2.6 64 POPCNT_X6 36.0 - 24.0 +14/=44/-2 60.00%
Stockfish 8 64 POPCNT_X6 - Houdini 4 Pro x64_St_Ct0_X6 46.5 - 13.5 +34/=25/-1 77.50%
Stockfish 8 64 POPCNT_X6 - Gull 3 x64 XP_X6 48.5 - 11.5 +38/=21/-1 80.83%

Incremental Time Control 2+2
Stockfish 8 64 POPCNT_X6 - Komodo 10.2 64-bit_X6_NOB 36.0 - 24.0 +16/=40/-4 60.00%
Stockfish 8 64 POPCNT_X6 - SugaR 2.6 64 POPCNT_X6 32.0 - 28.0 +8/=48/-4 53.33%
Stockfish 8 64 POPCNT_X6 - Houdini 4 Pro x64_St_Ct0_X6 47.0 - 13.0 +35/=24/-1 78.33%
Stockfish 8 64 POPCNT_X6 - Gull 3 x64 XP_X6 50.5 - 9.5 +41/=19/-0 84.17%

Score using 6 cores= 334.0 – 146.0 = 69.58%
480 Games = http://www.mediafire.com/file/yf39bvu732y0gy8/

8 real cores
Ponder: Off
GUI: Fritz 14
Book: Perfect 2014c – Limit 5 moves. Sedat –
No tablebases. No RTB used.
Hash 1024
Relative Speed: 53.22
Knodes per Second: 25.543

Fixed Time Control 4+0
Stockfish 8 64 BMI2_X8 - Komodo 10.2 64-bit_X8_NOB 39.5 - 20.5 +20/=39/-1 65.83%
Stockfish 8 64 BMI2_X8 - SugaR 2.6 64 BMI2_X8 34.0 - 26.0 +10/=48/-2 56.67%
Stockfish 8 64 BMI2_X8 - Houdini 4 Pro x64_Ct0_8 46.5 - 13.5 +33/=27/-0 77.50%
Stockfish 8 64 BMI2_X8 - Gull 3 x64 BMI2_X8 46.0 - 14.0 +32/=28/-0 76.67%

Incremental Time Control 2+2
Stockfish 8 64 BMI2_X8 - Komodo 10.2 64-bit_X8_NOB 39.5 - 20.5 +20/=39/-1 65.83%
Stockfish 8 64 BMI2_X8 - SugaR 2.6 64 BMI2_X8 33.5 - 26.5 +8/=51/-1 55.83%
Stockfish 8 64 BMI2_X8 - Houdini 4 Pro x64_Ct0_8 43.0 - 17.0 +26/=34/-0 71.67%
Stockfish 8 64 BMI2_X8 - Gull 3 x64 BMI2_X8 46.0 - 14.0 +32/=28/-0 76.67%

Score using 8 cores= 328.0 – 152.0 = 68.33%
480 Games= SF 8 (X8) = http://www.mediafire.com/file/a88zytuyyrjbmh6/

Global Score for Stockfish 8 after 960 games= 662.0 – 298.0 = 68.96%

Average Elo of rivals= 3202

Estimated Elo of Stockfish 8 after 960 games= 3335

Almost exactly the same Elo than his ‘brother’ Stockfish 081016 after 960 games (3.334)

Regards,

Tom.

Tomcass
Posts: 786
Joined: Sun Apr 16, 2006 7:09 pm

Re: For Tom- or Maybe Anyone Who Knows!!

Post by Tomcass » Mon Dec 05, 2016 10:51 am

TESTING STOCKFISH 8 ASMPD: 960 GAMES

6 real cores
Ponder: Off.
GUI: Fritz 14
Book: Perfect 2014c – Limit 5 moves. Sedat –
No tablebases. No RTB used.
Hash 512
Relative Speed: 29.54
Knodes per second: 14.177

Fixed Time Control= 4+0
SF8 ASMPD_X6 - Komodo 10.2 64-bit_X6_NOB 37.5 - 22.5 +18/=39/-3 62.50%
SF8 ASMPD_X6 - SugaR 2.6 64 POPCNT_X6 36.0 - 24.0 +14/=44/-2 60.00%
SF8 ASMPD_X6 - Houdini 4 Pro x64_St_Ct0_X6 44.5 - 15.5 +31/=27/-2 74.17%
SF8 ASMPD_X6 - Gull 3 x64 XP_X6 50.5 - 9.5 +41/=19/-0 84.17%

Incremental Time Control= 2+2
SF8 ASMPD_X6 - Komodo 10.2 64-bit_X6_NOB 36.0 - 24.0 +17/=38/-5 60.00%
SF8 ASMPD_X6 - SugaR 2.6 64 POPCNT_X6 36.0 - 24.0 +12/=48/-0 60.00%
SF8 ASMPD_X6 - Houdini 4 Pro x64_St_Ct0_X6 48.0 - 12.0 +36/=24/-0 80.00%
SF8 ASMPD_X6 - Gull 3 x64 XP_X6 47.5 - 12.5 +35/=25/-0 79.17%

480 Games= http://www.mediafire.com/file/rxtjpmzsqqlhjrp/
Score using 6 cores= 336.0 – 144.0 = 70.00%

8 real cores
Ponder: Off
GUI: Fritz 14
Book: Perfect 2014c – Limit 5 moves. Sedat –
No tablebases. No RTB used.
Hash 1024
Relative Speed: 53.22
Knodes per Second: 25.543

Fixed Time Control= 4+0
SF8 ASMPD_X8 - Komodo 10.2 64-bit_X8_NOB 36.5 - 23.5 +14/=45/-1 60.83%
SF8 ASMPD_X8 - SugaR 2.6 64 BMI2_X8 35.0 - 25.0 +10/=50/-0 58.33%
SF8 ASMPD_X8 - Houdini 4 Pro x64_Ct0_8 46.5 - 13.5 +33/=27/-0 77.50%
SF8 ASMPD_X8 - Gull 3 x64 BMI2_X8 48.0 - 12.0 +36/=24/-0 80.00%

Incremental Time Control= 2+2
SF8 ASMPD_X8 - Komodo 10.2 64-bit_X8_NOB 37.5 - 22.5 +18/=39/-3 62.50%
SF8 ASMPD_X8 - SugaR 2.6 64 BMI2_X8 35.0 - 25.0 +11/=48/-1 58.33%
SF8 ASMPD_X8 - Houdini 4 Pro x64_Ct0_8 45.0 - 15.0 +30/=30/-0 75.00%
SF8 ASMPD_X8 - Gull 3 x64 BMI2_X8 47.0 - 13.0 +35/=24/-1 78.33%

480 games: http://www.mediafire.com/file/1tfqy0pva8zmcyz/
Score using 8 cores = 330.5 – 149.5 = 68.85%

GLOBAL SCORE FOR SF8 ASMPD AFTER 960 GAMES = 666.5 – 293.5 = 69.43%
Average Elo of rivals= 3202
Estimated Elo of SF8 ASMPD 8 after 960 games= 3338

According to my latest two tests, the difference between STOCKFISH 8 (3335) and SF8 ASMPD (3338) doesn't seem to be very relevant. Only three ELO points, after 1.920 games.

Regards,

Tom.

Tomcass
Posts: 786
Joined: Sun Apr 16, 2006 7:09 pm

Re: For Tom- or Maybe Anyone Who Knows!!

Post by Tomcass » Sat Dec 24, 2016 6:27 pm

TESTING STOCKFISH 8: 1920 GAMES

6 real cores
Ponder: Off.
GUI: Fritz 14
Book: Perfect 2014c – Limit 5 moves. Sedat –
No tablebases. No RTB used.
Hash 512
Relative Speed: 29.54
Knodes per second: 14.177

Fixed Time Control 4+0
Stockfish 8 64 POPCNT_X6 - Houdini 5.01 Pro x64-popc_X6 58.0 - 62.0 +16/=84/-20 48.33%
Stockfish 8 64 POPCNT_X6 - SugaR 2.6 64 POPCNT_X6 62.0 - 58.0 +10/=104/-6 51.67%
Stockfish 8 64 POPCNT_X6 - Komodo 10.2 64-bit_X6_NOB 76.0 - 44.0 +40/=72/-8 63.33%
Stockfish 8 64 POPCNT_X6 - Deep Shredder 13 x64_X6 84.0 - 36.0 +50/=68/-2 70.00%

Score using 6 cores (4+0)= 280.0 – 200.0= 58.33%

Incremental Time Control 2+2
Stockfish 8 64 POPCNT_X6 - Houdini 5.01 Pro x64-popc_X6 59.0 - 61.0 +8/=102/-10 49.17%
Stockfish 8 64 POPCNT_X6 - SugaR 2.6 64 POPCNT_X6 63.5 - 56.5 +12/=103/-5 52.92%
Stockfish 8 64 POPCNT_X6 - Komodo 10.2 64-bit_X6_NOB 73.0 - 47.0 +38/=70/-12 60.83%
Stockfish 8 64 POPCNT_X6 - Deep Shredder 13 x64_X6 80.0 - 40.0 +43/=74/-3 66.67%

Score using 6 cores (2+2)= 275.5 – 204.5= 57.40%
960 Games = http://www.mediafire.com/file/hqo2ahf88euzij0/

8 real cores
Ponder: Off
GUI: Fritz 14
Book: Perfect 2014c – Limit 5 moves. Sedat –
No tablebases. No RTB used.
Hash 1024
Relative Speed: 53.22
Knodes per Second: 25.543

Fixed Time Control 4+0
Stockfish 8 64 BMI2_X8 - Houdini 5 Pro x64-pext_X8 59.0 - 61.0 +12/=94/-14 49.17%
Stockfish 8 64 BMI2_X8 - SugaR 2.6 64 BMI2_X8 66.0 - 54.0 +17/=98/-5 55.00%
Stockfish 8 64 BMI2_X8 - Komodo 10.2 64-bit_X8_NOB 68.5 - 51.5 +24/=89/-7 57.08%
Stockfish 8 64 BMI2_X8 - Deep Shredder 13 x64_X8 87.0 - 33.0 +58/=58/-4 72.50%

Score using 8 cores (4+0) = 280.5 – 199.5= 58.44%
480 Games http://www.mediafire.com/file/7qnvq7b7dgagc1c

Incremental Time Control 2+2
Stockfish 8 64 BMI2_X8 - Houdini 5 Pro x64-pext_X8 65.0 - 55.0 +19/=92/-9 54.17%
Stockfish 8 64 BMI2_X8 - SugaR 2.6 64 BMI2_X8 68.5 - 51.5 +22/=93/-5 57.08%
Stockfish 8 64 BMI2_X8 - Komodo 10.2 64-bit_X8_NOB 74.5 - 45.5 +32/=85/-3 62.08%
Stockfish 8 64 BMI2_X8 - Deep Shredder 13 x64_X8 85.0 - 35.0 +50/=70/-0 70.83%

Score using 8 cores (2+2) = 293.0 – 187.0= 61.04%
480 Games http://www.mediafire.com/file/7qnvq7b7dgagc1c/

GLOBAL SCORE AFTER 1920 GAMES: 1129.0 – 791.0= 58.80%
Average Elo of rivals= 3.263
Estimated ELO for Stosckfish 8 in this test= 3.325

ESTIMATED ELO FOR STOCKFISH 8 AFTER TWO TESTS WITH 2.880 GAMES: 3328 ELO POINTS.

My Short Rating List:

SF8 ASMPD = 3338
Stockfish 8= 3328
Houdini 5 Pro= 3319
Sugar2.6 = 3291
Komodo 10.2 = 3277
Stockfish 7 = 3.268
Deep Shredder = 3.174
Houdini 4 Pro= 3136
Gull 3= 3103

After this test, the difference between SF8 ASMPD and Stockfish 8 has increased to 10 points. Houdini 5 Pro is now only 9 Elo points below Stockfish 8.

Regards from Barcelona,
Tom.

Post Reply