Threads-Test - SF, Zappa, Komodo - 1 vs. 2, 4, 8, 16 Threads

[Ozymandias]

this is awesome and deeply intresting!!

Maybe Zappa would give 100% CPU utilization in future tourns?

[syzygy]

The diminishing returns that I think is about rating and not about time.
The weaker engine has lower rating so it has the potential to improve more from 1 thread to many threads.

Give SF very little time and it will have a very low rating.

Suppose you are correct and SF gained much less from time doubling / speed doubling (those are the same!!) than some other engine, independent of time control. Then either SF would have to be outrageously much stronger than the other engine at ultrashort time controls (which it is not), or SF would be much weaker than the other engine at long time controls (which it is not). So you are not correct.

An extreme example is when the stronger engine play perfect and in this case it is obvious that the stronger engine cannot improve by more threads.

Wrong example. SF is far, far, far, far, far, far, far from perfect at ultrashort time controls.

[Martin Thoresen]

Seems you are luckier than me with a 16-core Xeon, because there Zappa with 16 threads are crashing very often.

[fastgm]

All tests runs with ponder=off
The test 1 against 32 Threads with 3000 games is rather a matter of time.

[fastgm]

Zappa with 16 threads crashed 3 times within 3000 games on my machine.

[Martin Thoresen]

Zappa with 16 threads crashed 3 times within 3000 games on my machine.

Yeah that is maybe expected, but still way better than maybe 30% on my machine.

[Uri Blass]

The diminishing returns that I think is about rating and not about time.
The weaker engine has lower rating so it has the potential to improve more from 1 thread to many threads.

Give SF very little time and it will have a very low rating.

Suppose you are correct and SF gained much less from time doubling / speed doubling (those are the same!!) than some other engine, independent of time control. Then either SF would have to be outrageously much stronger than the other engine at ultrashort time controls (which it is not), or SF would be much weaker than the other engine at long time controls (which it is not). So you are not correct.

An extreme example is when the stronger engine play perfect and in this case it is obvious that the stronger engine cannot improve by more threads.

Wrong example. SF is far, far, far, far, far, far, far from perfect at ultrashort time controls.

I believe that SF has a better search algorithm in the first point and
it is a reason that it can earn more elo (or at least the same elo) from doubling the speed with 1 core at Super fast time control.

I am not sure if it is fair to compare SMP implementation for different searches because it is possible that for some simple search it is easier to get bigger speed improvement so
I still think that the only fair comparison is when you start from the same elo.

Even if Zappa 8 cores is equivalent to being 6 times faster than Zappa 1 core when Stockfish 8 cores is equivalent to being 4 times faster than Stockfish 1 core it does not mean that Zappa implementation is better from my point of view because
it is possible that zappa's relative stupid search make it easier to get speed improvement from more cores.

Edit:Note that I do not claim that zappa implementation is not superior and from my point of view earning more elo from more cores when you start from the same elo is better.

My guess is that zappa earns more elo from more cores from the same elo starting point but it is not proved.

[syzygy]

Even if Zappa 8 cores is equivalent to being 6 times faster than Zappa 1 core when Stockfish 8 cores is equivalent to being 4 times faster than Stockfish 1 core it does not mean that Zappa implementation is better from my point of view because
it is possible that zappa's relative stupid search make it easier to get speed improvement from more cores.

I am afraid that from your point of view it is never possible to say anything at all.

[Laskos]

The diminishing returns that I think is about rating and not about time.
The weaker engine has lower rating so it has the potential to improve more from 1 thread to many threads.

Give SF very little time and it will have a very low rating.

Suppose you are correct and SF gained much less from time doubling / speed doubling (those are the same!!) than some other engine, independent of time control. Then either SF would have to be outrageously much stronger than the other engine at ultrashort time controls (which it is not), or SF would be much weaker than the other engine at long time controls (which it is not). So you are not correct.

An extreme example is when the stronger engine play perfect and in this case it is obvious that the stronger engine cannot improve by more threads.

Wrong example. SF is far, far, far, far, far, far, far from perfect at ultrashort time controls.

I believe that SF has a better search algorithm in the first point and
it is a reason that it can earn more elo (or at least the same elo) from doubling the speed with 1 core at Super fast time control.

I am not sure if it is fair to compare SMP implementation for different searches because it is possible that for some simple search it is easier to get bigger speed improvement so
I still think that the only fair comparison is when you start from the same elo.

Even if Zappa 8 cores is equivalent to being 6 times faster than Zappa 1 core when Stockfish 8 cores is equivalent to being 4 times faster than Stockfish 1 core it does not mean that Zappa implementation is better from my point of view because
it is possible that zappa's relative stupid search make it easier to get speed improvement from more cores.

Edit:Note that I do not claim that zappa implementation is not superior and from my point of view earning more elo from more cores when you start from the same elo is better.

My guess is that zappa earns more elo from more cores from the same elo starting point but it is not proved.

SMP effective speed-up (usually TTD, but I don't know for Zappa) is larger with time control. So, say we equal Zappa's strength to that of SF by giving Zappa 10 times more time. It will have an effective speed-up even larger relatively to what is presented here.

[Laskos]

The diminishing returns that I think is about rating and not about time.
The weaker engine has lower rating so it has the potential to improve more from 1 thread to many threads.

Give SF very little time and it will have a very low rating.

Suppose you are correct and SF gained much less from time doubling / speed doubling (those are the same!!) than some other engine, independent of time control. Then either SF would have to be outrageously much stronger than the other engine at ultrashort time controls (which it is not), or SF would be much weaker than the other engine at long time controls (which it is not). So you are not correct.

An extreme example is when the stronger engine play perfect and in this case it is obvious that the stronger engine cannot improve by more threads.

Wrong example. SF is far, far, far, far, far, far, far from perfect at ultrashort time controls.

I believe that SF has a better search algorithm in the first point and
it is a reason that it can earn more elo (or at least the same elo) from doubling the speed with 1 core at Super fast time control.

I am not sure if it is fair to compare SMP implementation for different searches because it is possible that for some simple search it is easier to get bigger speed improvement so
I still think that the only fair comparison is when you start from the same elo.

Even if Zappa 8 cores is equivalent to being 6 times faster than Zappa 1 core when Stockfish 8 cores is equivalent to being 4 times faster than Stockfish 1 core it does not mean that Zappa implementation is better from my point of view because
it is possible that zappa's relative stupid search make it easier to get speed improvement from more cores.

Edit:Note that I do not claim that zappa implementation is not superior and from my point of view earning more elo from more cores when you start from the same elo is better.

My guess is that zappa earns more elo from more cores from the same elo starting point but it is not proved.

SMP effective speed-up (usually TTD, but I don't know for Zappa) is larger with time control. So, say we equal Zappa's strength to that of SF by giving Zappa 10 times more time. It will have an effective speed-up even larger relatively to what is presented here.

I tested SMP efficiency at same _strength_ for SF and Zappa Mexico II, giving ~11 times more time to Zappa at time-to-depth test. The results are pretty conclusive
100 position repeated 4 times, the average time to depth on 100 positions:

Code: Select all

Depth=20
SF 1  thread: 12:40
SF 4 threads:  4:22

Effective SF speed-up on 4 threads: 2.90

Code: Select all

Depth=15
Zappa Mexico II 1  thread: 140:20
Zappa Mexico II 4 threads:  43:45

Effective Zappa speed-up on 4 threads: 3.21

So, at same Elo, Zappa scales better on SMP than SF, and seemingly artifact of lower Elo engine improvement for Zappa is not supported.