How many concurrent games for testing?

[RayLeeIsmay]

Hi everyone, it's my first post!

I've put together my first basic engine and am starting to play with search/evaluation and testing. Currently I'm testing first with WAC for a quick sanity check and then by running 1k games (each) against an older version of my engine and TSCP. I know this isn't nearly enough but I'm working on a laptop so my resources are limited.

My CPU (Intel i7-9750H CPU @ 2.60GHz ) has 6 cores and 12 threads. I'm testing with cutechess. How many concurrent games do you think I should run? I'd like to be testing significantly more games, but I'm not sure how many concurrent games my processor could reasonably handle.

Thanks for your help!

[algerbrex]

Hi Rayleigh, welcome to TalkChess.

I do all of my testing on a laptop as well (Ryzen 4700u with 8 cores, 8 threads, 8 GB RAM), and I always limit the concurrency to no more than the number of cores I have (so a maximum of 8 games in parallel in this case). I've found this has let me get away with doing fairly quick testing while not being too much of a strain on my laptop.

My laptop will get quite hot doing this however, so I've also invested in a small desktop fan to aid the internal fan to keep the laptop fairly cool.

[j.t.]

My laptop will get quite hot doing this however, so I've also invested in a small desktop fan to aid the internal fan to keep the laptop fairly cool.

One thing I sometimes do when my notebook is under load, is to put it on two thick books, so that the fan intake on the bottom has more room to suck up fresh air.

Regarding the number of concurrent games, when using my personal machine and I don't plan doing anything else on the PC and I test single thread chess engine versions, then I run the same number of games as there are number of threads that the CPU supports. I am not sure if it would be better to no rely on hyperthreading, but it works.

[Ajedrecista]

Hello:

Welcome to TalkChess!

A similar question was posted at TalkChess in 2017:

cutechess-cli concurrency

I am neither a programmer nor a tester, so take my advice like a grain of salt, but I have read many times that the advisable maximum concurrency is physical cores - 1, that is, 5 in your case. The reason is leaving one core to the OS without doing extra stuff like web browsing or whatever.

Another thing to worry about is cooling, as said in the first reply. A laptop is not maybe the best choice to run under full CPU load for long periods of time, but if you still want it, make sure to provide a good cooling, even lifting the laptop over two books (minimal contact between the books and the laptop) to allow the air circulating:

Code: Select all

   LAPTOP
BOOK    BOOK

I see that the books option is popular and already posted. Other option is to put four glass shots downwards (one under each corner of the laptop) over a stable table or on the floor, to achieve a similar effect than the books.

And remeber, my advice of 5 is a maximum that can be reduced to 4 or 3 in extreme cases like heat waves, always doing nothing more than chess testing while chess testing.

Regards from Spain.

Ajedrecista.

[RayLeeIsmay]

Thanks everyone for your suggestions. I've also been concerned about heat, so I've been using the rather ridiculous solution of keeping my laptop on top of a box fan, which is on top of two books.

Code: Select all

                 [Laptop]
         [--------Fan----------]
    [book]                    [book]
            

[algerbrex]

My laptop will get quite hot doing this however, so I've also invested in a small desktop fan to aid the internal fan to keep the laptop fairly cool.

One thing I sometimes do when my notebook is under load, is to put it on two thick books, so that the fan intake on the bottom has more room to suck up fresh air.

Good point. At some point I really ought to invest in a dedicated cooling pad I can use, although the desktop fan has been surprisingly effective in keeping my laptop cool. When I have it running on full throttle, the surface isn't even hot to the touch, more warm.

[dangi12012]

I have switched to github actions some time ago.
Firstly you can still use your own hardware in the background - but by default you get some CPU hours for free!
https://docs.github.com/en/actions

What is making this perfect - is that this stuff runs on commit on your private repo and you instantly get the feedback if your commit improved strength when running a cutechess task there.
You can pick if you run it on github's hardware or your own.

[jdart]

You should be using SPRT to tell if you your game test results are statistically significant.
I think it is almost impossible to run enough games on a 6-core laptop to have most changes pass SPRT in a reasonable time.
I am afraid you need a lot of fast cores for that.