Threads factor: Komodo, Houdini, Stockfish and Zappa

[kasinp]

Thank you for sharing these interesting results. There is little data around this topic and it is becoming more and more relevant.

PK

[Modern Times]

But that is not the point of the experiment. This tells us about the upper limit of scalability, which is useful to know. In addition, it tells us how that upper limits suffers from addition of cores. For instance, Houdini starts to have problems after exactly 16 cores. Before that, it is among the best.

Miguel

Yes I agree, these tests are very interesting. Thanks to Andreas for running them.

[Laskos]

Interesting information but the target of chess programs is not to search more nodes but to earn playing strength.

Nodes are not proportional to playing strength and I guess that for the same engine,
the same number of nodes with 1 thread is better than the same number of nodes with many threads.

But that is not the point of the experiment. This tells us about the upper limit of scalability, which is useful to know. In addition, it tells us how that upper limits suffers from addition of cores. For instance, Houdini starts to have problems after exactly 16 cores. Before that, it is among the best.

Miguel

Miguel, as you know

1/ MP NPS are scaling with time or depth, so at 200s per position they will be different (and probably higher in most of cases) from 20s per position

2/ NPS of Rybka Cluster and Jonny on 2,000+ cores scaled almost linearly with the number of cores, and they lost to Xeon Houdini and Junior respectively. So, their effective speed-up is way below what NPS shows. And effective speed-up from MP will scale with time too, increasing with time per position.

[zullil]

It would be interesting to see how the newer SF dev versions are doing compared to SF DD.

Here are some data from the very latest Stockfish:

Code: Select all

Threads          NPS          NPS/(NPS for 1 thread)
--------------------------------------------------------
     1            1 279 478                 1.00
    
     2            2 732 441                 2.14

     4            5 176 548                 4.05

     8            9 432 155                 7.37

    16           17 073 731                13.34

Each line was produced by invoking the following from a command line, with threads = 1, 2, 4, 8, 16 successively.

Code: Select all

./stockfish bench 1024 threads 20 benchfile.txt time

Thus the hash size was 1GB, and each position in benchfile.txt was searched for 20 seconds. In this experiment, benchfile.txt consisted of 15 copies of the opening position in chess.

[michiguel]

Interesting information but the target of chess programs is not to search more nodes but to earn playing strength.

Nodes are not proportional to playing strength and I guess that for the same engine,
the same number of nodes with 1 thread is better than the same number of nodes with many threads.

But that is not the point of the experiment. This tells us about the upper limit of scalability, which is useful to know. In addition, it tells us how that upper limits suffers from addition of cores. For instance, Houdini starts to have problems after exactly 16 cores. Before that, it is among the best.

Miguel

Miguel, as you know

1/ MP NPS are scaling with time or depth, so at 200s per position they will be different (and probably higher in most of cases) from 20s per position

2/ NPS of Rybka Cluster and Jonny on 2,000+ cores scaled almost linearly with the number of cores, and they lost to Xeon Houdini and Junior respectively. So, their effective speed-up is way below what NPS shows. And effective speed-up from MP will scale with time too, increasing with time per position.

Absolutely. It would be interesting to repeat this at 200s (I do not believe it would be needed to run it 5 times, one will be enough, and it could done with 2,4,6 etc). I bet all curves will go up.

Of course, the ceiling for jonny or R. was much higher than reality.

Miguel

[Joerg Oster]

These are NPS. Hard to tell strength-wise, or effective speed-up. Time to depth (TTD) won't help too much either, as even SF with Joona's patch widens a bit, without talking of Komodo.

What makes you think that SF now widens a bit?
Afaik, there is no change in the search algorithm for SMP ...

[Laskos]

These are NPS. Hard to tell strength-wise, or effective speed-up. Time to depth (TTD) won't help too much either, as even SF with Joona's patch widens a bit, without talking of Komodo.

What makes you think that SF now widens a bit?
Afaik, there is no change in the search algorithm for SMP ...

I posted right after the release of Joona's patch:
http://www.talkchess.com/forum/viewtopi ... 6&start=11

[Joerg Oster]

These are NPS. Hard to tell strength-wise, or effective speed-up. Time to depth (TTD) won't help too much either, as even SF with Joona's patch widens a bit, without talking of Komodo.

What makes you think that SF now widens a bit?
Afaik, there is no change in the search algorithm for SMP ...

I posted right after the release of Joona's patch:
http://www.talkchess.com/forum/viewtopi ... 6&start=11

Sorry, I missed that one.
Kind of strange, because as far as I understand Joona's patch, there is no change in the search algorithm. Idle threads can now try to actively join a split point. Maybe this is due to more search overhead?

[Laskos]

These are NPS. Hard to tell strength-wise, or effective speed-up. Time to depth (TTD) won't help too much either, as even SF with Joona's patch widens a bit, without talking of Komodo.

What makes you think that SF now widens a bit?
Afaik, there is no change in the search algorithm for SMP ...

I posted right after the release of Joona's patch:
http://www.talkchess.com/forum/viewtopi ... 6&start=11

Sorry, I missed that one.
Kind of strange, because as far as I understand Joona's patch, there is no change in the search algorithm. Idle threads can now try to actively join a split point. Maybe this is due to more search overhead?

Maybe, I don't know, I was so surprised that I then set alpha, beta to 0.01 instead of 0.05 (LLR 4.59 instead of 2.94) to SPRT, and H1=10 was again accepted, so it's only 1% a false positive. Fixed depth:

Code: Select all

    Program                            Score      %      Elo    +   -   Draws

  1 SF 4 threads                  : 2831.0/5425  52.2      8    6   6   60.6 %
  2 SF 1 thread                   : 2594.0/5425  47.8     -8    6   6   60.6 %

15 +/- 6 Elo points widening 2SD from 1 to 4 threads.

[yolin]

For scalability, node count is only one aspect though. Here is a suggestion to test the true scalability of the engines. Consider the following match conditions

For a fixed engine (selfplay)
1 thread 32 min/game vs 2 threads 16 min/game
1 thread 32 min/game vs 4 threads 8 min/game
1 thread 32 min/game vs 8 threads 4 min/game
1 thread 32 min/game vs 16 threads 2 min/game
1 thread 32 min/game vs 32 threads 1 min/game

The elo difference for the different threads should give a pretty good indication of how well an engine scales. Perfect scaling would give 50% score for all games.