Testing Stockfish 11-03-13. 480 Games.

[carldaman]

You cannot really test 2 engines against each other while you have LP on with one and not the other. That would be similar to testing 2 engines with one of them on 10% or 15% faster hardware and then assuming that the engine is outperforming the other. If LP are to be turned ON then they should be done for both engines.

For what its worth, LP are really not worth the trouble as they quickly deragment your memory. And unless you have a good memory deragmenter and realize when your memory needs mainenance, then it is just too much of a headache. This is especially true if you are engine testing as you really have no clue when LP are working properly or when they are not. I do see how they can be useful if you absolutely need the highest performance out of your system for long term analysis ... but even then you could simply improve your cooling and overclock your CPU for easier and better results.

... by the way, M.Ansari, perhaps this link will be useful for you -and for all members that join this excellent forum-:

http://www.cruxis.com/chess/manual/inde ... ersion.htm

You will see that in fact the situation is very different -in fact the opposite- to the one that you assumed as right: Houdini 3.0 uses always Large Pages, providing they are available.

Regards,

Tom.

Can Houdini use large pages if the memory is too fragmented ? Would that hurt Houdini's performance relative to no-LP engines, since there is no way to tell Houdini not to use large pages ?

Regards,
CL

[M ANSARI]

You cannot really test 2 engines against each other while you have LP on with one and not the other. That would be similar to testing 2 engines with one of them on 10% or 15% faster hardware and then assuming that the engine is outperforming the other. If LP are to be turned ON then they should be done for both engines.

For what its worth, LP are really not worth the trouble as they quickly deragment your memory. And unless you have a good memory deragmenter and realize when your memory needs mainenance, then it is just too much of a headache. This is especially true if you are engine testing as you really have no clue when LP are working properly or when they are not. I do see how they can be useful if you absolutely need the highest performance out of your system for long term analysis ... but even then you could simply improve your cooling and overclock your CPU for easier and better results.

... by the way, M.Ansari, perhaps this link will be useful for you -and for all members that join this excellent forum-:

http://www.cruxis.com/chess/manual/inde ... ersion.htm

You will see that in fact the situation is very different -in fact the opposite- to the one that you assumed as right: Houdini 3.0 uses always Large Pages, providing they are available.

Regards,

Tom.

Ticking the LP square will not necessarily mean that LP will now work. If the memory is fragmented LP will NOT work even if you have clicked it on. I learned this the hard way when testing engine tournaments ... it will work for the first few games and then it will stop working without giving any signs that the LP stopped working. That is what I am trying to say, it just adds a factor that if not properly checked will give wrong data. For testing engines it is best to keep it off. If you want to run one engine for analysis and need every ounce of performance then you might find it useful.

[Tomcass]

Stockfish 191013 With Large Pages (Peterpan’s Compile) = 480 Games.

i7 980 3.33 Ghz.
6 real cores
Ponder: Off.
GUI: Fritz 12
Book: Perfect 2012c
No tablebases. No RTB used.
Large Pages Allowed
Hash 512
Relative Speed: 28.66
Knodes per second: 13.759

Time Control_4+0

201310SF191013ST66_4+0_120games_LP_PP 2013

Stockfish 191013SL 64 SSE4.2L - Critter 1.6a 64-bitX6NOB 27.0 - 13.0 +19/=16/-5 67.50%
Stockfish 191013SL 64 SSE4.2L - Houdini 3 Pro x64_6 21.0 - 19.0 +10/=22/-8 52.50%
Stockfish 191013SL 64 SSE4.2L - Komodo 6 64-bitx6NOB 22.0 - 18.0 +11/=22/-7 55.00%

Time Control_2+2

201310SF191013ST66_2+2_120games_LP_PP 2013

Stockfish 191013SL 64 SSE4.2L - Critter 1.6a 64-bitX6_NOB 22.5 - 17.5 +11/=23/-6 56.25%
Stockfish 191013SL 64 SSE4.2L - Houdini 3 Pro x64_6 19.5 - 20.5 +9/=21/-10 48.75%
Stockfish 191013SL 64 SSE4.2L - Komodo 6 64-bitx6NOB 22.5 - 17.5 +12/=21/-7 56.25%


240 Games X 6 Cores = http://www.mediafire.com/?7tb50t8db8zzl76
Score using 6 cores = 134.5 – 105.5 = 56.04%

i7 975 3.33 Ghz.
4 real cores
Ponder: Off.
GUI: Fritz 12
Book: Perfect 2012c
No tablebases. No RTB used.
Large Pages Allowed
Hash 256
Relative Speed: 20.62
Knodes per second: 9.899

Time Control_4+0

Stockfish 191013SL 64 SSE4.2L - Critter 1.6a 64-bitnob_4 23.5 - 16.5 +13/=21/-6 58.75%
Stockfish 191013SL 64 SSE4.2L - Houdini 3 Pro x64_4 20.5 - 19.5 +13/=15/-12 51.25%
Stockfish 191013SL 64 SSE4.2L - Komodo 6 64-bitx4_NOB 24.0 - 16.0 +12/=24/-4 60.00%

Time Control_2+2

Stockfish 191013SL 64 SSE4.2L - Critter 1.6a 64-bitnob_4 24.5 - 15.5 +15/=19/-6 61.25%
Stockfish 191013SL 64 SSE4.2L - Houdini 3 Pro x64_4 18.0 - 22.0 +9/=18/-13 45.00%
Stockfish 191013SL 64 SSE4.2L - Komodo 6 64-bitx4_NOB 25.5 - 14.5 +14/=23/-3 63.75%

240 Games X 4 cores = http://www.mediafire.com/?bpvovy4o4dl0xjn
Scores using 4 cores = 136.0 – 104.0 = 56.67%

Segmenting the result by Time Control:

Fixed = 138.0-102.0 = 57.50%
Incremental = 132.5-107.5 = 55.21%

Global Score: 270.5 – 209.5= 56.35%
Against Critter 1.6a (3093): 60.94% Houdini 3.0 Pro (3172): 49.37% Komodo 6 (3154): 58.75%

Average ELO of opponents= 3.140
Estimated ELO Performance= 3.184


Because of a business trip to Latin-America I will stop testing for about one month. I hope that my computers will be grateful to me after 8 month working day and night non-stop with my tests. By the way I can not imagine where will be the level of Stockfish in one month.

Enjoy great computer chess, my friends.

Best regards from Barcelona.

Tom.

[Kohflote]

Hi Tom,

Have a smooth and safe trip. Looking forward to your return and further testing of SF

Best wishes,
Koh, Kah Huat

[Tomcass]

I wrote this sentence about 4 weeks ago:

… I can not imagine where Stockfish will be in one month.

Here you have the answer:

Stockfish 111113SL 64 SSE4.2L With Large Pages (Ipman’s Compile) = 480 Games.

i7 980 3.33 Ghz.
6 real cores
Ponder: Off.
GUI: Fritz 12
Book: Perfect 2012c
No tablebases. No RTB used.
Large Pages Allowed
Hash 512
Relative Speed: 28.66
Knodes per second: 13.759

Time Control_4+0

Stockfish 111113SL 64 SSE4.2L - Critter 1.6a 64-bitX6_NOB 27.5 - 12.5 +20/=15/-5 68.75%
Stockfish 111113SL 64 SSE4.2L - Houdini 3 Pro x64_6 25.5 - 14.5 +18/=15/-7 63.75%
Stockfish 111113SL 64 SSE4.2L - Komodo 6 64-bitx6NOB 23.5 - 16.5 +13/=21/-6 58.75%

Time Control_2+2

Stockfish 111113SL 64 SSE4.2L - Critter 1.6a 64-bitX6_NOB 28.5 - 11.5 +17/=23/-0 71.25%
Stockfish 111113SL 64 SSE4.2L - Houdini 3 Pro x64_6 20.5 - 19.5 +9/=23/-8 51.25%
Stockfish 111113SL 64 SSE4.2L - Komodo 6 64-bitx6NOB 21.0 - 19.0 +14/=14/-12 52.50%

240 Games X 6 Cores = http://www.mediafire.com/?k02ltjuo34mckgy
Score using 6 cores = 146.5 – 93.5 = 61.04%

i7 975 3.33 Ghz.
4 real cores
Ponder: Off.
GUI: Fritz 12
Book: Perfect 2012c
No tablebases. No RTB used.
Large Pages Allowed
Hash 256
Relative Speed: 20.62
Knodes per second: 9.899

Time Control_4+0

Stockfish 111113SL 64 SSE4.2I - Critter 1.6a 64-bitnob_4 28.0 - 12.0 +18/=20/-2 70.00%
Stockfish 111113SL 64 SSE4.2I - Houdini 3 Pro x64_4 22.0 - 18.0 +13/=18/-9 55.00%
Stockfish 111113SL 64 SSE4.2I - Komodo 6 64-bitx4_NOB 24.0 - 16.0 +14/=20/-6 60.00%

Time Control_2+2

Stockfish 111113SL 64 SSE4.2I - Critter 1.6a 64-bitnob_4 23.5 - 16.5 +14/=19/-7 58.75%
Stockfish 111113SL 64 SSE4.2I - Houdini 3 Pro x64_4 26.0 - 14.0 +14/=24/-2 65.00%
Stockfish 111113SL 64 SSE4.2I - Komodo 6 64-bitx4_NOB 27.5 - 12.5 +17/=21/-2 68.75%


240 Games X 4 cores = http://www.mediafire.com/?8tkmgcybhh37r8z
Scores using 4 cores = 151.0 - 89.0 = 62.92%

Segmenting the result by Time Control:

Fixed = 150.5 – 89.5 = 62.71%
Incremental = 147.0 – 93.0 = 61.25%

Global Score: 297.5 – 182.5= 61.98%
Against Critter 1.6a (3093): 67.19% Houdini 3.0 Pro (3172): 58.75% Komodo 6 (3154): 60.00%

Average ELO of opponents= 3.140
Estimated ELO Performance= 3.224

According to this test, this Ipman’s compile of Stockfish, using Large Pages but not tablebases, is:

52 ELO points above Houdini 3.0 Pro.
70 ELO points above Komodo 6 and
131 ELO points above Critter 1.6a

I can not find any adjective to explain this incredible score. If you have a bit of minuts spare, please give a look, randomly, to some games.

Best regards from Barcelona.

Tom.

[gladius]

I can not find any adjective to explain this incredible score. If you have a bit of minuts spare, please give a look, randomly, to some games.

Welcome back Tom. Thanks for the wonderful tests once again! I think SF might have gotten a bit lucky in these results, but still, it's nice to see .

[beram]

I can not find any adjective to explain this incredible score. If you have a bit of minuts spare, please give a look, randomly, to some games.

Welcome back Tom. Thanks for the wonderful tests once again! I think SF might have gotten a bit lucky in these results, but still, it's nice to see .

Indeed amazing.
Btw what can be the positive contribution of these large pages allowed ?
I had read that it doesnt make much difference
But if it does it is perhaps unfair testing. Could you or Gary explain this ?

thx in advance Bram

[Kohflote]

Hi Tom,

Welcome back and glad to see your testing again . Two questions, please:

(1) where do get IPman's version?
(2) what do you think the performance of SF same version without LP?

Best wishes,
Koh, Kah Huat

[Tomcass]

Hi, Gary!

Thanks for your kind words of welcome back. In fact I agree with you that SF has been a bit lucky in this test. But believe me, I have seen some impressive games in this test. Anyway I do not expect for SF to repeat such an incredible score in the next few ... weeks.

Best regards,

Tom.

[Tomcass]

I can not find any adjective to explain this incredible score. If you have a bit of minuts spare, please give a look, randomly, to some games.

Welcome back Tom. Thanks for the wonderful tests once again! I think SF might have gotten a bit lucky in these results, but still, it's nice to see .

Indeed amazing.
Btw what can be the positive contribution of these large pages allowed ?
I had read that it doesnt make much difference
But if it does it is perhaps unfair testing. Could you or Gary explain this ?

thx in advance Bram

Hi, Bram.

I think that this test is not unfair at all. I try to test the best version of each engine. If Houdini 3.0 Pro uses Large Pages -according to Mr. Houdart website- why SF can not use them?. I will be delighted to us also LP with Komodo or Critter when possible. (Not till now, I think).

I can not give an exact reference of the importance of LP. My best 'guesstimate', -important!- in my computers and under my testing conditions, is between 5 and 6 ELO points improvement. Perhaps under another testing and computing environment the result can be different.

Regards,

Tom.