SMP experts advice needed

[lucasart]

I know that some of you are experts on SMP, and would like your advice on my testing methodology.

- I have N=8 CPU (i7), and would like to run lots of games, for testing purposes (with Stockfish)
- SMP implementation in chess engines cannot scale linearly, so I cannot hope that SF will be 8x faster with 8 CPU (perhaps 6-7?)
- So it's best to leave the parrallelisation to the tournament manager, which is cutechess-cli (the "-concurrency" option)
=> so far, you agree ?

- I've heard that it's not a good idea to run N=8 games concurrently, but rather N-1 concurrent games
=> Is that true ? Intuitively it would seem logical, but has there been any experiment to demonstrate it ? (I'm thinking of hacking cutechess-cli to make it spit out stats on average and stdev of NPS per engine for example).

- I saw an hyper-threading option in my BIOS.
=> Should I enable it ? Do you reckon it would increase my aberage NPS in such an experiment ? Or would it decrease the stdev(NPS), which is also important as concurrency pollution introduces another source of noise in the operation ?

PS: Obviously, cutechess-cli uses threads of its own, just like a lot of different processes running in the backgroupd of my OS. But all that should be very negligible (I'm using Linux 3.5.0 + LXDE, all *very* lightweight).

Thank you

[bob]

I know that some of you are experts on SMP, and would like your advice on my testing methodology.

- I have N=8 CPU (i7), and would like to run lots of games, for testing purposes (with Stockfish)
- SMP implementation in chess engines cannot scale linearly, so I cannot hope that SF will be 8x faster with 8 CPU (perhaps 6-7?)
- So it's best to leave the parrallelisation to the tournament manager, which is cutechess-cli (the "-concurrency" option)
=> so far, you agree ?

- I've heard that it's not a good idea to run N=8 games concurrently, but rather N-1 concurrent games
=> Is that true ? Intuitively it would seem logical, but has there been any experiment to demonstrate it ? (I'm thinking of hacking cutechess-cli to make it spit out stats on average and stdev of NPS per engine for example).

- I saw an hyper-threading option in my BIOS.
=> Should I enable it ? Do you reckon it would increase my aberage NPS in such an experiment ? Or would it decrease the stdev(NPS), which is also important as concurrency pollution introduces another source of noise in the operation ?

PS: Obviously, cutechess-cli uses threads of its own, just like a lot of different processes running in the backgroupd of my OS. But all that should be very negligible (I'm using Linux 3.5.0 + LXDE, all *very* lightweight).

Thank you

I see no reason to run n-1 games with N cores. I test on a cluster with 70 nodes, each node has 8 cpus, and I play 8 games per node at a time. I have tested with 1, 2, 4, 6, 7 and 8 games at a time and found absolutely no significant difference between them. 8 games at a time is the fastest, and that is what I use all the time...

[Dirt]

I know that some of you are experts on SMP, and would like your advice on my testing methodology.

I am not an expert, so don't assume what I say is true.

- I saw an hyper-threading option in my BIOS.
=> Should I enable it ? Do you reckon it would increase my aberage NPS in such an experiment ? Or would it decrease the stdev(NPS), which is also important as concurrency pollution introduces another source of noise in the operation ?

I would say you should turn on hyperthreading, as long as are running each engine on a single core. If you want to test with two or more cores per engine, it is probably safer to turn it off.

[lucasart]

I know that some of you are experts on SMP, and would like your advice on my testing methodology.

- I have N=8 CPU (i7), and would like to run lots of games, for testing purposes (with Stockfish)
- SMP implementation in chess engines cannot scale linearly, so I cannot hope that SF will be 8x faster with 8 CPU (perhaps 6-7?)
- So it's best to leave the parrallelisation to the tournament manager, which is cutechess-cli (the "-concurrency" option)
=> so far, you agree ?

- I've heard that it's not a good idea to run N=8 games concurrently, but rather N-1 concurrent games
=> Is that true ? Intuitively it would seem logical, but has there been any experiment to demonstrate it ? (I'm thinking of hacking cutechess-cli to make it spit out stats on average and stdev of NPS per engine for example).

- I saw an hyper-threading option in my BIOS.
=> Should I enable it ? Do you reckon it would increase my aberage NPS in such an experiment ? Or would it decrease the stdev(NPS), which is also important as concurrency pollution introduces another source of noise in the operation ?

PS: Obviously, cutechess-cli uses threads of its own, just like a lot of different processes running in the backgroupd of my OS. But all that should be very negligible (I'm using Linux 3.5.0 + LXDE, all *very* lightweight).

Thank you

I see no reason to run n-1 games with N cores. I test on a cluster with 70 nodes, each node has 8 cpus, and I play 8 games per node at a time. I have tested with 1, 2, 4, 6, 7 and 8 games at a time and found absolutely no significant difference between them. 8 games at a time is the fastest, and that is what I use all the time...

OK, thank you for debunking the myth!

The concern I had is a risk-reward reasoning:
- you go from 7 to 8 cores, and you (on average) multiply your total NPS by 8/7 (+14.3%)
- now if the noise that accompanies it (using the same computer to browse internet and do basic things that aren't computationally intensive, but still) happens to increase the varianve by something larger than 8/7. Then it means that 8 CPU is effectively less efficient than 7 (because to achieve the same precision you need more games, and worse, you don't know or measure that idiosyncratic variance).
- anyway, that's all theoretical masturbation, and what matters is empirical reality. So I'll trust your judgment if you say that the results were no different regardless of the number of concurrent games.

[bob]

I know that some of you are experts on SMP, and would like your advice on my testing methodology.

- I have N=8 CPU (i7), and would like to run lots of games, for testing purposes (with Stockfish)
- SMP implementation in chess engines cannot scale linearly, so I cannot hope that SF will be 8x faster with 8 CPU (perhaps 6-7?)
- So it's best to leave the parrallelisation to the tournament manager, which is cutechess-cli (the "-concurrency" option)
=> so far, you agree ?

- I've heard that it's not a good idea to run N=8 games concurrently, but rather N-1 concurrent games
=> Is that true ? Intuitively it would seem logical, but has there been any experiment to demonstrate it ? (I'm thinking of hacking cutechess-cli to make it spit out stats on average and stdev of NPS per engine for example).

- I saw an hyper-threading option in my BIOS.
=> Should I enable it ? Do you reckon it would increase my aberage NPS in such an experiment ? Or would it decrease the stdev(NPS), which is also important as concurrency pollution introduces another source of noise in the operation ?

PS: Obviously, cutechess-cli uses threads of its own, just like a lot of different processes running in the backgroupd of my OS. But all that should be very negligible (I'm using Linux 3.5.0 + LXDE, all *very* lightweight).

Thank you

I see no reason to run n-1 games with N cores. I test on a cluster with 70 nodes, each node has 8 cpus, and I play 8 games per node at a time. I have tested with 1, 2, 4, 6, 7 and 8 games at a time and found absolutely no significant difference between them. 8 games at a time is the fastest, and that is what I use all the time...

OK, thank you for debunking the myth!

The concern I had is a risk-reward reasoning:
- you go from 7 to 8 cores, and you (on average) multiply your total NPS by 8/7 (+14.3%)
- now if the noise that accompanies it (using the same computer to browse internet and do basic things that aren't computationally intensive, but still) happens to increase the varianve by something larger than 8/7. Then it means that 8 CPU is effectively less efficient than 7 (because to achieve the same precision you need more games, and worse, you don't know or measure that idiosyncratic variance).
- anyway, that's all theoretical masturbation, and what matters is empirical reality. So I'll trust your judgment if you say that the results were no different regardless of the number of concurrent games.

What you are describing is NOT what you asked. You want no more than 8 active threads/processes if you have 8 cores. Whether that is 8 chess engines, or 4 chess engines and 4 something else's doesn't matter. My test machines play chess and nothing else, so there is no reason to hold one or more cores in reserve. If you test on your normal machine, you already have too much random noise and I'd consider testing on a machine that I use for other things to be essentially worthless, because the extra load will influence games in unpredictable ways...

[hgm]

- I saw an hyper-threading option

Are you French?

[mcostalba]

I'll trust your judgment

I never trust the judgement especially that of other people than me.

I have not done a full study, but I have many empirical anecdotes I found during my tests that show that testing with concurrency > 1 adds noise.

But the point is deeper than that. Unfortunately current ELO estimators do not take in account the different noise models: for bayes elo (but the others are the same) the info you can get from 1000 games is the same regardless if the games where hyper bullet 2" games ran on a computer while you were browsing and watching a movie at the same time or they were high quality, longer TC games on a dedicated machine.

The noise level is completely different in both cases but current ELO estimators, amazingly, do not take in account that and give the same LOS independently from the testing conditions (never understood why, probably because people working on them do not have a good background on information exchange over a noisy channel).

[lucasart]

- I saw an hyper-threading option

Are you French?

Yes, but what's that got to do with hyper-threading ?

[lucasart]

I'll just settle for the middle-ground: use concurrency = N-1. So as to leave 1 CPU for the OS processes, browsing the web etc. (I'm NOT doing computationally intensive things on this machine, just browsing forums like this or doing basic things like spreadsheets in gnumeric, but certainly not watching videos).

[hgm]

Are you French?

Yes, but what's that got to do with hyper-threading ?

Because of your name I had classified you in the back of my mind as German.

What gave it away was that you made a spelling mistake that is only natural to an English speaker with a heavy French accent, which does not pronounce the 'h': "an hyper-threading option".

I used to fake similar errors (like "an hydrogen atom") whenever I had to write a very negative report on a paper I had to referee, to miguide the authors about my identity.