parallel search speed measurement

[Laskos]

I would be interested to see a problem solving test done in parallel with time allotted inversely proportional to threads.

E.g. given 100 very hard problems (e.g. Der brillante Schachzug 100 problems)

Give 100 problems 32 seconds each with 1 thread
Give 100 problems 16 seconds each with 2 threads
Give 100 problems 8 seconds each with 4 threads
Give 100 problems 4 seconds each with 8 threads
Give 100 problems 2 seconds each with 16 threads
Give 100 problems 1 seconds each with 32 threads (or whatever you have available)

About one hour per test for the 1 thread and 32 seconds for the 32 threads.

In theory, the smaller thread count would do better because of having no SMP loss. But I suspect that the heavy thread counts would do better in the same time.

The reason is that I suspect parallel searching may actually find the answers faster due to SMP uncertainty, but it is only a guess.

Bob is testing time-to-depth, not time-to-solution or number of solutions. If the engine doesn't widen or narrowing its search, like Crafty 25, then time-to-depth is the best and easiest measure. If it uses, for example widening Lazy-SMP, like Komodo or Stockfish, then your proposal is interesting. I considered it since awhile, because it's easier to use a testsuite than to run many multithreaded games at long time control. Based on Arasan 19 testsuite with 1 minute per position on 1 thread, I concluded a rather low speedup 1 -> 4 cores of 2.6 for Komodo and 2.4 for Stockfish dev. I am not sure why the numbers are so low. Even Arasan 19 seems too hard for Crafty at say 1 minute 1 thread test, it solves only about 20% of positions. In fact, I am amazed at how weak Crafty 25, a year 2016 formerly top engine, tactically is. On WM testusite is solves about 50% of positions n 1 minute 1-thread search, worse than Shredder 8 of 2003. But the parallel search in Crafty 25 seems to work well, on time-to-depth I am getting about 3.2 speedup 1 -> 4 cores, better than Komodo with 2.6 and Stockfish dev. with 2.4 on Arasan 19.

[Dann Corbit]

Lazy SMP gives better Elo, but clearly does not get as deep as quickly.
The new Stockfish loses several plies in the same amount of time on an hour search compared to the traditional search version.

[Laskos]

Lazy SMP gives better Elo, but clearly does not get as deep as quickly.
The new Stockfish loses several plies in the same amount of time on an hour search compared to the traditional search version.

I don't understand why Lazy SMP gives better ELO. According to this test with Arasan 19 testsuite, the effective speed-up is worse than that of Crafty 25. Depth is irrelevant, as for Komodo and Stockfish I just counted the number of solutions and time to solutions. It is possible that tactical testsuite approach is wrong.