Testing Stockfish 11-03-13. 480 Games.

[Tomcass]

STOCKFISH 4 _ 480 GAMES

Timestamp: 1376982085

Bench signature is: 4132374

i7 980 3.33 Ghz.
6 real cores
Ponder: Off.
GUI: Fritz 12
Book: Perfect 2012c
No tablebases
Hash 512
Relative Speed: 28.66
Knodes per second: 13.759


Time Control_4+0_120games

Stockfish 4 64 SSE4.2x6 - Critter 1.6 64-bitx6_nob 22.5 - 17.5 +8/=29/-3 56.25%
Stockfish 4 64 SSE4.2x6 - Deep Rybka 4.1 SSE42 x64 (x6) 27.5 - 12.5 +18/=19/-3 68.75%
Stockfish 4 64 SSE4.2x6 - Houdini 3 Pro x64_6 17.5 - 22.5 +11/=13/-16 43.75%

Time Control _2+2_120games

Stockfish 4 64 SSE4.2x6 - Critter 1.6 64-bitx6_nob 22.0 - 18.0 +11/=22/-7 55.00%
Stockfish 4 64 SSE4.2x6 - Deep Rybka 4.1 SSE42 x64 (x6) 25.0 - 15.0 +16/=18/-6 62.50%
Stockfish 4 64 SSE4.2x6 - Houdini 3 Pro x64_6 18.0 - 22.0 +9/=18/-13 45.00%

240 Games = http://www.mediafire.com/?56o0cweo1o54jyq
Score using 6 cores = 132.5-107.5 = 55.21%

i7 975 3.33 Ghz.
4 real cores
Ponder: Off.
GUI: Fritz 12
Book: Perfect 2012c
No tablebases
Hash 256
Relative Speed: 20.62
Knodes per second: 9.899

Time Control_4+0_120games

Stockfish 4 64 SSE4.2x5 - Critter 1.6a 64-bitnob_4 19.5 - 20.5 +10/=19/-11 48.75%
Stockfish 4 64 SSE4.2x5 - Deep Rybka 4 x64 (x4) 26.0 - 14.0 +18/=16/-6 65.00%
Stockfish 4 64 SSE4.2x5 - Houdini 3 Pro x64_4 18.5 - 21.5 +10/=17/-13 46.25%

Time Control _2+2_120games

Stockfish 4 64 SSE4.2x5 - Critter 1.6a 64-bitnob_4 23.5 - 16.5 +11/=25/-4 58.75%
Stockfish 4 64 SSE4.2x5 - Deep Rybka 4 x64 (x4) 28.0 - 12.0 +17/=22/-1 70.00%
Stockfish 4 64 SSE4.2x5 - Houdini 3 Pro x64_4 16.0 - 24.0 +4/=24/-12 40.00%

240 Games = http://www.mediafire.com/?ifbnpgn4jw3lkki
Score Using 4 Cores = 131.5-108.5 = 54.79%
Segmenting by Time Control:

Fixed = 54.79%
Incremental = 55.21%

Global Score: 264.0–216.0 = 55.00%
Against Critter 1.6: 54.69% Deep Rybka 4: 66.55% Houdini 3.0: 43.75%
Estimated ELO Performance: 3129

According to this test, SF 4 is roughly:
43 ELO points below Houdini 3.0 Pro. (3172)
2 ELO points below Komodo 5 1r2 (3131)
36 ELO points better than Critter 1.6 (3093) and
113 ELO points better than Rybka 4 (3016)

Let's test it against Komodo 5 1r2.

Regards,

Tom.

[beram]

STOCKFISH 4 _ 480 GAMES

..

Fixed = 54.79%
Incremental = 55.21%

Global Score: 264.0–216.0 = 55.00%
Against Critter 1.6: 54.69% Deep Rybka 4: 66.55% Houdini 3.0: 43.75%
Estimated ELO Performance: 3129

According to this test, SF 4 is roughly:
43 ELO points below Houdini 3.0 Pro. (3172)
2 ELO points below Komodo 5 1r2 (3131)
36 ELO points better than Critter 1.6 (3093) and
113 ELO points better than Rybka 4 (3016)

Let's test it against Komodo 5 1r2.

Regards,

Tom.

Glad, that would be interesting.
My guess, according to my tests and the testing by Martin Wijngaarden, it will be a tiny bit stronger (0-10 ELO)

grts Bram

[Tomcass]

STOCKFISH 4 _ 480 GAMES

..

Fixed = 54.79%
Incremental = 55.21%

Global Score: 264.0–216.0 = 55.00%
Against Critter 1.6: 54.69% Deep Rybka 4: 66.55% Houdini 3.0: 43.75%
Estimated ELO Performance: 3129

According to this test, SF 4 is roughly:
43 ELO points below Houdini 3.0 Pro. (3172)
2 ELO points below Komodo 5 1r2 (3131)
36 ELO points better than Critter 1.6 (3093) and
113 ELO points better than Rybka 4 (3016)

Let's test it against Komodo 5 1r2.

Regards,

Tom.

Glad, that would be interesting.
My guess, according to my tests and the testing by Martin Wijngaarden, it will be a tiny bit stronger (0-10 ELO)

grts Bram

Hi, Bram.

A difference of 0-10 ELO points can be explained by statistical noise. I am glad to know that my results are very coincident with Martin's and yours, which I consider to be very reliable.

In fact, I am testing -400 games- SF4 against Komodo 5.1r2 and the result is not bad at all. Tomorrow I will post this test.

Kind regards,

Tom.

[Vinvin]

STOCKFISH 4 _ 480 GAMES
...Global Score: 264.0–216.0 = 55.00%
Against Critter 1.6: 54.69% Deep Rybka 4: 66.55% Houdini 3.0: 43.75%
Estimated ELO Performance: 3129
...
Tom.

In your last results, I see some perf at around "3120" and some perf around "3220" ... is there a glitch somewhere ?

[Tomcass]

STOCKFISH 4 _ 480 GAMES
...Global Score: 264.0–216.0 = 55.00%
Against Critter 1.6: 54.69% Deep Rybka 4: 66.55% Houdini 3.0: 43.75%
Estimated ELO Performance: 3129
...
Tom.

In your last results, I see some perf at around "3120" and some perf around "3220" ... is there a glitch somewhere ?

Yes, Vincent. I am very sorry. There is a mistake I have noticed only recently. The right ELO estimate for Stockfish in my tests is the one starting by 31xx rather than 32xx. I am not able to edit past posts and change this value, as I did in another forum.

My top ELO estimate is for Houdini 3.0 Pro with 3.172, so that anything above this value is obviously wrong. For Stockfish 4 the Estimated ELO score has been already fixed: 3129 in my first test.

Thank you very much for your comment pointing out my mistake, Vincent.

Regards,

Tom.

[Tomcass]

Stockfish 4 - Komodo 5.1r2 : 400 GAMES

i7 980 3.33 Ghz.
6 real cores
Ponder: Off.
GUI: Fritz 12
Book: Perfect 2012c
No tablebases
Hash 512
Relative Speed: 28.66
Knodes per second: 13.759

201308Stockfish4_Komodo_4+0_120games 2013

1 Komodo 5.1r2 64-bitx6nob +25/=53/-22 51.50% 51.5/100
2 Stockfish 4 64 SSE4.2x6 +22/=53/-25 48.50% 48.5/100

201308Stockfish4_Komodo_2+2_120games 2013

1 Komodo 5.1r2 64-bitnob +23/=61/-16 53.50% 53.5/100
2 Stockfish 4 64 SSE4.2x6 +16/=61/-23 46.50% 46.5/100

200 Games = http://www.mediafire.com/?y0r22bhf86rzfmt
Score using 6 cores = Komodo 5.1r2 – Stockfish 4 105.0 – 95.0

i7 975 3.33 Ghz.
4 real cores
Ponder: Off.
GUI: Fritz 12
Book: Perfect 2012c
No tablebases
Hash 256
Relative Speed: 20.62
Knodes per second: 9.899

20130824_SF4_Komodo_4+0 2013

1 Stockfish 4 64 SSE4.2x5 +28/=52/-20 54.00% 54.0/100
2 Komodo 5.1r2 64-bitnob +20/=52/-28 46.00% 46.0/100

20130824_SF4_Komodo_2+2 2013


1 Komodo 5.1r2 64-bitnob +26/=55/-19 53.50% 53.5/100
2 Stockfish 4 64 SSE4.2x5 +19/=55/-26 46.50% 46.5/100

200 Games = http://www.mediafire.com/?qxnfaan9n928fdd

Score using 4 cores = Komodo 5.1r2 – Stockfish 4 99.5 – 100.5

Global Score after 400 Games = Komodo 5.1r2 – Stockfish 4 204.5 – 195.5
51.125% - 48.875%

Regards,

Tom.

[ernest]

Time Control_4+0_120games
Stockfish 4 64 SSE4.2x6 - Houdini 3 Pro x64_6 17.5 - 22.5 +11/=13/-16 43.75%

Time Control _2+2_120games
Stockfish 4 64 SSE4.2x6 - Houdini 3 Pro x64_6 18.0 - 22.0 +9/=18/-13 45.00%

Time Control_4+0_120games
Stockfish 4 64 SSE4.2x5 - Houdini 3 Pro x64_4 18.5 - 21.5 +10/=17/-13 46.25%

Time Control _2+2_120games
Stockfish 4 64 SSE4.2x5 - Houdini 3 Pro x64_4 16.0 - 24.0 +4/=24/-12 40.00%

I also was not able to see Stockfish 4 beat Houdini 3 in a match (220 games, 2'+1")

[Tomcass]

Stockfish 230813 = 480 GAMES.

Bench: 4729333 Timestamp: 1377175148

i7 980 3.33 Ghz.
6 real cores
Ponder: Off.
GUI: Fritz 12
Book: Perfect 2012c
No tablebases
Hash 512
Relative Speed: 28.66
Knodes per second: 13.759

Time Control_4+0

Stockfish 230813 64 SSE4.2_ - Critter 1.6 64-bitx6_nob 20.0 - 20.0 +10/=20/-10 50.00%
Stockfish 230813 64 SSE4.2_ - Deep Rybka 4.1 SSE42 x64 28.5 - 11.5 +20/=17/-3 71.25%
Stockfish 230813 64 SSE4.2_ - Houdini 3 Pro x64_6 23.0 - 17.0 +13/=20/-7 57.50%

Time Control_2+2

Stockfish 230813 64 SSE4.2_ - Critter 1.6 64-bitx6_nob 23.0 - 17.0 +13/=20/-7 57.50%
Stockfish 230813 64 SSE4.2_ - Deep Rybka 4.1 SSE42 x64 24.5 - 15.5 +14/=21/-5 61.25%
Stockfish 230813 64 SSE4.2_ - Houdini 3 Pro x64_6 17.0 - 23.0 +8/=18/-14 42.50%

240 Games X6 = http://www.mediafire.com/?ylo5ygvzoos7e5x
Score using 6 cores = 136.0-104.0 = 56.67%

i7 975 3.33 Ghz.
4 real cores
Ponder: Off.
GUI: Fritz 12
Book: Perfect 2012c
No tablebases
Hash 256
Relative Speed: 20.62
Knodes per second: 9.899

Time Control_4+0

Stockfish 230813 64 SSE4.2_ - Critter 1.6a 64-bitnob_4 23.0 - 17.0 +10/=26/-4 57.50%
Stockfish 230813 64 SSE4.2_ - Deep Rybka 4 x64 (x4) 28.5 - 11.5 +21/=15/-4 71.25%
Stockfish 230813 64 SSE4.2_ - Houdini 3 Pro x64_4 17.5 - 22.5 +5/=25/-10 43.75%

Time Control_2+2

Stockfish 230813 64 SSE4.2_ - Critter 1.6a 64-bitnob_4 25.5 - 14.5 +17/=17/-6 63.75%
Stockfish 230813 64 SSE4.2_ - Deep Rybka 4 x64 (x4) 23.5 - 16.5 +16/=15/-9 58.75%
Stockfish 230813 64 SSE4.2_ - Houdini 3 Pro x64_4 21.0 - 19.0 +11/=20/-9 52.50%

240 Games X4 = http://www.mediafire.com/?i6d77o9o5879oix
Score using 4 cores = 139.0-101.0 = 57.92%


Segmenting by Time Control:

Fixed = 58.54%
Incremental = 56.04%

Global Score: 275.0–205.0 = 57.29%
New global best score for Stockfish

Against Critter 1.6: 57.19% Deep Rybka 4: 65.62% Houdini 3.0: 49.06% (New best score against Houdini 3.0)
Estimated ELO Performance: 3145

Only as a reference, according to my tests –and obviously subject to any statistical noise- this version is about 16 ELO points stronger than Stockfish 4, 14 ELO points stronger than Komodo 5.1r2 and only 27 ELO points below Houdini 3.0 Pro.

This result is simply impressive. I will test the new development Stockfish version to confirm -or otherwise- the validity of this score.

Smiling regards from Barcelona.

Tom.

[Kohflote]

Dear Tom,

Could I clarify the version of Stockfish in your last test, please? In your post, you wrote 230813 version, Timestamp: 1377175148.

Do you mean the version of Date: Thu Aug 22 14:39:08 2013 +0200
Timestamp: 1377175148, Author: homoSapiensSapiens? I could not find 230813 version at http://abrok.eu/stockfish/ site.

Thank you & regards,
Kah Huat

[Tomcass]

Dear Kah Huat,

The version I tested is Bench: 4729333 Timestamp: 1377175148.

Although the version is dated Thu Aug 22 you will see that the name that appears when you create the new engine is Stockfish 230813 64 SSE4.2. I don't know the reason.

Kind regards, my friend.

Tom.