Stockfish 8 - Double time control vs. 2 threads

Discussion of computer chess matches and engine tournaments.

Moderators: hgm, Rebel, chrisw

IWB
Posts: 1539
Joined: Thu Mar 09, 2006 2:02 pm

Re: Stockfish 8 - Quadruple time control vs. 4 threads

Post by IWB »

Ferdinand Mosca wrote a GREAT tool to check my database. I have to thank him very much for this!

THANKS Ferdinand, great work, much appreciated!!!!

At 5+3 Ponder on for the TOP16 it looks like this:

Code: Select all

A. Players list:

  1 Andscacs 0.88
  2 Booot 6.1
  3 Chiron 3.01
  4 Critter 1.6a
  5 Equinox 3.30
  6 Fizbo 1.8
  7 Fritz 15
  8 Ginkgo 1.8
  9 Gull 3
 10 Hannibal 1.7
 11 Houdini 5
 12 Jonny 8.00
 13 Komodo 10.2
 14 Protector 1.9.0
 15 Shredder 13
 16 Stockfish 8

B. Ponder hit and miss table:

 nr player                     :     hit     miss   hit%   games   pts%

  1 Komodo 10.2                :  107261    72022   59.8    3300   52.1
  2 Houdini 5                  :  101643    72338   58.4    3300   52.1
  3 Shredder 13                :  105672    76146   58.1    3300   52.1
  4 Fritz 15                   :  113329    81693   58.1    3300   52.1
  5 Critter 1.6a               :  113195    81988   58.0    3300   52.1
  6 Stockfish 8                :   91870    66839   57.9    3300   52.1
  7 Protector 1.9.0            :  104780    76961   57.7    3300   52.1
  8 Chiron 3.01                :  103214    76467   57.4    3300   52.1
  9 Equinox 3.30               :  108174    80217   57.4    3300   52.1
 10 Booot 6.1                  :  109026    81087   57.3    3300   52.1
 11 Ginkgo 1.8                 :  107277    80097   57.3    3300   52.1
 12 Andscacs 0.88              :  114635    86850   56.9    3300   52.1
 13 Gull 3                     :  114069    88119   56.4    3300   52.1
 14 Jonny 8.00                 :  103241    82396   55.6    3300   52.1
 15 Fizbo 1.8                  :  108686    88565   55.1    3300   52.1
 16 Hannibal 1.7               :  104148    85945   54.8    3300   52.1

Average hit %  : 57.2
File           : TOP16L.pgn
Date           : 2016-11-30 11:47:33
Elapsed (sec)  : 6435.2
The important number is the average hit rate of 57.2%!

The ratio from 8 to 16 core is 9.55/6.62 = 1.5701. That means that the cutoff for more efficient (aka "better") games is 16 cores. 2x8core Ponder ON gives better result (qualitative better games) in average over 1x16core Ponder OFF! In other words, Ponder OFF games, played on computers with 16 cores or more, are a waste of resources!
And keep in mind this is only 5+3. If you run longer games with a higher draw rate you will end up most likely with a higher Ponder HIT rate and a higher Ponder ON advantage with lower core numbers!

WOW
Ingo

PS: Iwill check for the TOP3 engines only.
PPS: To produce equaly good games to my 5 + 3 someone has to ply 471s + 4.71s games with Ponder off (~ 8m + 4s) on equal hw
IWB
Posts: 1539
Joined: Thu Mar 09, 2006 2:02 pm

Re: Stockfish 8 - Quadruple time control vs. 4 threads

Post by IWB »

The TOP 4:

Code: Select all

A. Players list:

  1 Houdini 5
  2 Komodo 10.2
  3 Shredder 13
  4 Stockfish 8

B. Ponder hit and miss table:

 nr player                     :     hit     miss   hit%   games   pts%

  1 Houdini 5                  :   23568    14522   61.9     660   58.5
  2 Komodo 10.2                :   23891    14729   61.9     660   58.5
  3 Stockfish 8                :   20926    13495   60.8     660   58.5
  4 Shredder 13                :   20957    14037   59.9     660   58.5

Average hit %  : 61.1
File           : TOP4L.pgn
Date           : 2016-11-30 12:29:14
Elapsed (sec)  : 342.7
61.1%, now we are on a 14 core CPU :-)

12 cores equals a Ponder hit rate of 63.9%. I think it is save to say that starting from 12 cores, high-end engine matches, especially with a longer time control, should be run with Ponder ON on 2x6 cores to get the maximum performance.

Ingo
fastgm
Posts: 818
Joined: Mon Aug 19, 2013 6:57 pm

Stockfish 8 - 8x time control vs. 8 threads

Post by fastgm »

Stockfish 8
1 thread (5 minutes + 3 seconds) vs. 8 threads (37.5 seconds + 0.375 seconds)

Code: Select all

    Program                                   Elo    +   -   Games   Score   Av.Op.  Draws
 -------------------------------------------------------------------------------------------
  1 Stockfish 8 - 1 thread     5' + 3"      : 3020   5   5   3000    55.8 %   2980   82.0 %
  2 Stockfish 8 - 8 threads 37.5" + 0.375"  : 2980   5   5   3000    44.2 %   3020   82.0 %


Wins   = 444
Draws  = 2459
Losses = 97
Av.Op. Elo = 3000

Result     : 1673.5/3000 (+444,=2459,-97)
Perf.      : 55.8 %
Margins    :
 68 %      : (+  0.4,-  0.4 %) -> [ 55.4, 56.2 %]
 95 %      : (+  0.7,-  0.7 %) -> [ 55.1, 56.5 %]
 99.7 %    : (+  1.1,-  1.1 %) -> [ 54.7, 56.9 %]

Elo        : 3040
Margins    :
 68 %      : (+  3,-  3) -> [3038,3043]
 95 %      : (+  5,-  5) -> [3035,3046]
 99.7 %    : (+  8,-  8) -> [3033,3048]


Games        :   3000 (finished)

White Wins   :    398 (13.3 %)
Black Wins   :    143 ( 4.8 %)
Draws        :   2459 (82.0 %)

White Perf.  : 54.2 %
Black Perf.  : 45.8 %


Individual statistics:

1 Stockfish 8 - 1 thread     5' + 3"      : 3020  3000 (+444,=2459,- 97), 55.8 %
2 Stockfish 8 - 8 threads 37.5" + 0.375"  : 2980  3000 (+ 97,=2459,-444), 44.2 %


Engine                                  Depth       Time   Games     Moves  Average
-----------------------------------------------------------------------------------
Stockfish 8 - 1 thread     5' + 3"      28.75  349:46:03    3000    187772    6.71
Stockfish 8 - 8 threads 37.5" + 0.375"  24.97   45:56:46    3000    187946    0.88
User avatar
Laskos
Posts: 10948
Joined: Wed Jul 26, 2006 10:21 pm
Full name: Kai Laskos

Re: Stockfish 8 - 8x time control vs. 8 threads

Post by Laskos »

fastgm wrote:Stockfish 8
1 thread (5 minutes + 3 seconds) vs. 8 threads (37.5 seconds + 0.375 seconds)

Code: Select all

    Program                                   Elo    +   -   Games   Score   Av.Op.  Draws
 -------------------------------------------------------------------------------------------
  1 Stockfish 8 - 1 thread     5' + 3"      : 3020   5   5   3000    55.8 %   2980   82.0 %
  2 Stockfish 8 - 8 threads 37.5" + 0.375"  : 2980   5   5   3000    44.2 %   3020   82.0 %


Wins   = 444
Draws  = 2459
Losses = 97
Av.Op. Elo = 3000

Result     : 1673.5/3000 (+444,=2459,-97)
Perf.      : 55.8 %
Margins    :
 68 %      : (+  0.4,-  0.4 %) -> [ 55.4, 56.2 %]
 95 %      : (+  0.7,-  0.7 %) -> [ 55.1, 56.5 %]
 99.7 %    : (+  1.1,-  1.1 %) -> [ 54.7, 56.9 %]

Elo        : 3040
Margins    :
 68 %      : (+  3,-  3) -> [3038,3043]
 95 %      : (+  5,-  5) -> [3035,3046]
 99.7 %    : (+  8,-  8) -> [3033,3048]


Games        :   3000 (finished)

White Wins   :    398 (13.3 %)
Black Wins   :    143 ( 4.8 %)
Draws        :   2459 (82.0 %)

White Perf.  : 54.2 %
Black Perf.  : 45.8 %


Individual statistics:

1 Stockfish 8 - 1 thread     5' + 3"      : 3020  3000 (+444,=2459,- 97), 55.8 %
2 Stockfish 8 - 8 threads 37.5" + 0.375"  : 2980  3000 (+ 97,=2459,-444), 44.2 %


Engine                                  Depth       Time   Games     Moves  Average
-----------------------------------------------------------------------------------
Stockfish 8 - 1 thread     5' + 3"      28.75  349:46:03    3000    187772    6.71
Stockfish 8 - 8 threads 37.5" + 0.375"  24.97   45:56:46    3000    187946    0.88
Another extremely important result. Thanks, Andreas, this thread is the most important empirical result for Lazy SMP.
Earlier I wrote predictions:
Very interesting observation. And in line to Amdahl's fit I did for Komodo 9.3 using Andreas' results:
http://www.talkchess.com/forum/viewtopi ... 4&start=45
Andreas posts very important, hardcore results which use huge CPU time.

From your observation, the predictions would be:

1 --> 8 threads: 6.1
1 --> 16 threads: 9.55
1 --> 32 threads: 13.4

That is somewhat higher than the old YBW numbers.
Let's see how 1 --> 8 threads prediction stands, which assumes 95.5% efficiency for Amdahl's law as observed by Milos:
From another Andreas' excellent test on doubling time:
http://www.talkchess.com/forum/viewtopi ... 84&start=0
we see that these 3 doublings in time control are 112+101+93 = 306 ELO points. The difference here for 1 --> 8 threads compared to 8x (3 doubling) in time is 40 ELO points.
So, 1 - 40/306 ~ 0.869 is the exponent of 8, therefore the speed-up 1 --> 8 threads is 8^0.869 ~ 6.09, exactly as predicted. This is AMAZING. The precision is even better than statistical error margins. So, as of now, it all fits perfectly Amdahl's law for Stockfish 8 effective speed-up:

  • Effective Speed-Up = 1 / (1 - 0.955 + 0.955/n_cores)
With more CPU and NUMA the speed-up will probably deteriorate even further than Amdahl's law predicts.
mjlef
Posts: 1494
Joined: Thu Mar 30, 2006 2:08 pm

Re: Stockfish 8 - 8x time control vs. 8 threads

Post by mjlef »

If accurate, with an infinite number of cores, the limited speedup is 22.2222222..
ernest
Posts: 2041
Joined: Wed Mar 08, 2006 8:30 pm

Re: Stockfish 8 - Double time control vs. 2 threads

Post by ernest »

Laskos wrote:Isn't the answer in Andreas' experiment:

http://www.talkchess.com/forum/viewtopi ... 84&start=0
It certainly is, Kai, thanks !

I had completely missed that thread in October...

Concerning the Dmitri Gusev's bizarre result with asmFish, I just answered to Mark Lefler in
http://www.talkchess.com/forum/viewtopi ... 149#697149