Stockfish "Use Sleeping Threads" Test

[zullil]

Is the following statement reasonable?

Suppose that for each position and each fixed amount of search time, 16 threads reaches a higher depth than 8 threads. Then 16 threads is likely to perform at least as well as 8 threads, as measured by winning chess games.

[bob]

Is the following statement reasonable?

Suppose that for each position and each fixed amount of search time, 16 threads reaches a higher depth than 8 threads. Then 16 threads is likely to perform at least as well as 8 threads, as measured by winning chess games.

Absolutely. But incredibly unlikely. To the point of "winning-the-lottery" type probability.

And the same rule still applies. Not just one run but several. But if you can improve the depth, then the thing will be stronger. Just watch out for flying pigs while doing this test.

[zullil]

Is the following statement reasonable?

Suppose that for each position and each fixed amount of search time, 16 threads reaches a higher depth than 8 threads. Then 16 threads is likely to perform at least as well as 8 threads, as measured by winning chess games.

Just watch out for flying pigs while doing this test.

[MikeB]

...
while having both hyperthreading and Use Sleeping Threads enabled gives a speedup of about 10% compared to having no hyperthreading. (I have checked that my machine runs the 8 threads on 8 distinct physical cores. i.e., no hyperthreading.)

...

tha's what I saw too - it works best with both enabled ~ 10% gain I3, two physcial cores, 4 logical cores, Windows 7 , 64 bit.

Mike

[zullil]

I understand that SMP is quite variable. When you say that the "test is no good" do you mean more than "the results are statistically insignificant?"

The interesting question, therefore, isn't whether the average N/s with 16 threads is higher than with 8 threads, but whether Stockfish is on average able to complete deeper searches in a given amount of time with 16 threads.

Thanks. So my test should have been to give each version of the engine some fixed amount of time to search and record the depth that it reached. (And to do this with a collection of positions and repeat these a large number of times to ensure significance of the results.)

No. It is much easier to pick a set of positions that you feel are representative. Opening, middlegame and endgame positions. Some tactical. Some positional. Some with one best move. Some with many nearly equally good moves.

Search them to fixed depth, but for each position choose a depth that makes it use significant time. Say at least one minute per position. As you run them, you will notice how wildly the times will fluctuate for many positions. You may well find a few that are very consistent. you don't need to run those dozens of times. But the more variability you get, the more runs you need to compute a reasonable mean.

Oh, I see. Rather than trying to see which thread setting gives the larger value for depth/time, we can just as well see which setting gives the lower value for time/depth. So my mistake was to focus on NPS rather than total time taken to search to a fixed depth.

[ernest]

So my mistake was to focus on NPS rather than total time taken to search to a fixed depth.

Yes, but my experience is that MP non-reproducibility is much worse for time (and nodes searched) to a fixed depth, than for NPS...

[zullil]

So my mistake was to focus on NPS rather than total time taken to search to a fixed depth.

Yes, but my experience is that MP non-reproducibility is much worse for time (and nodes searched) to a fixed depth, than for NPS...

Yes, the variance in nps is much smaller than in total time. Thus the need for multiple searches of each position for in order to obtain a statistically significant estimate of time/depth. Since I was mistakenly focusing on nps, I was downplaying this issue.

I've learned (or been reminded about) a lot in this thread. Thanks to all who responded.

[IQ]

Is the following statement reasonable?

Suppose that for each position and each fixed amount of search time, 16 threads reaches a higher depth than 8 threads. Then 16 threads is likely to perform at least as well as 8 threads, as measured by winning chess games.

Absolutely. But incredibly unlikely. To the point of "winning-the-lottery" type probability.

And the same rule still applies. Not just one run but several. But if you can improve the depth, then the thing will be stronger. Just watch out for flying pigs while doing this test.

I disagree here. Even a higher displayed depth in a fixed time means nothing. It could very well be that through the non deterministic nature of the smp, hash table interaction and the high selectivity of modern programs that a higher depth is reached without playing stronger. The best test in my mind would be the TIME to SOLUTION of positions with known best moves (or as an approximation the MOVE where a reasonable large sample of engines agree on as depth goes to infinity). If you average time to solution over a reasonable number of positions (whose estimates themselves should be averages of multiple runs) you should be fine. Don't let yourself be fooled by depth and nodes programs display, in a parallel world and with modern selective programs their informative value is relative.

[mcostalba]

I've learned (or been reminded about) a lot in this thread. Thanks to all who responded.

My very trivial and rude opinion here is the following.

We have two ways to proceeding:


1) Continue to "learning" discussing interesting theories in the forum

2) Verify what really happens doing a test on real games with proper conditions, well yes, maybe TC should be increased a bit say to 30"+0.1

Sorry to be so pragmatic

[Houdini]

2) Verify what really happens doing a test on real games with proper conditions, well yes, maybe TC should be increased a bit say to 30"+0.1

Even 30"+0.1" is way too fast for this test.
For a 8-thread v 4-thread match I recommend an average move time of at least 3 to 5 seconds. For example 2'+2" would be probably be fine.

Robert