Time controls for testing for 40/4

[AndrewGrant]

Up until now my testing procedure has been highly unorganized. My tests are, at best, a 100game test versus another engine in 40moves in 4 minutes time controls. This is problematic because 100games at this time control takes hours and hours (Was testing in Arena). I now have cutechess-cli set up and a dozen or so cores I can be playing games on.

My question is, what time control is faster than 40/4, but can still have the results somewhat transfer?

[jeffreyan11]

It depends. When you're still early in development you don't care too much about scaling, etc. and you can do quick tests with very short time controls and very few games. I used to test with 100 games at 15 sec / game.

Once the engine gets stronger changes are harder to detect since each one gives less elo, so you need thousands of games. Then you base time controls on what the change is. For most evaluation changes I test at 3+0.02s and confirm at 15+0.05s. For search changes I do 15 sec or 60 sec per game. If you're more concerned with scaling to LTC you can do a 5+ min / game confirmation test.

TLDR: 40/4 is way too long, 15 sec / game or less is a much more efficient use of resources useless you have tons of supercomputer clusters.

[cdani]

If you're more concerned with scaling to LTC you can do a 5+ min / game confirmation test.

Beware that the test being at 5 min does not change anything. You must do a lot of games anyway if you are testing for a little improvement.

[Robert Pope]

Up until now my testing procedure has been highly unorganized. My tests are, at best, a 100game test versus another engine in 40moves in 4 minutes time controls. This is problematic because 100games at this time control takes hours and hours (Was testing in Arena). I now have cutechess-cli set up and a dozen or so cores I can be playing games on.

My question is, what time control is faster than 40/4, but can still have the results somewhat transfer?

I test at 6s+0.15s. That lets me run 2000 games overnight on 3 cores. I find that my program does not react as well to extremely short increments, so I don't go any faster than this.