What's the fastest time control you can effectively test at?

Discussion of chess software programming and technical issues.

Moderators: hgm, Harvey Williamson, bob

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
Post Reply
jordanbray
Posts: 51
Joined: Mon Aug 11, 2014 1:01 am

What's the fastest time control you can effectively test at?

Post by jordanbray » Sat May 30, 2015 1:28 pm

I've been running more tests to tune my engine, and have now gotten in the habbit of playing lots of games with certain features disabled, to see if anything was actually hurting the engine.

Turns out, at 40/.5+.01 time control, many of the evaluation features have performed poorly. However, I'm becoming a bit concerned that the time control is too fast.

However, running at this short of a time control enables me to run many more tests (46500 games overnight, with 31 (!) builds of the engine).

What time control do you most often test at, and what penalties exist for tuning to these fast time controls?

Ferdy
Posts: 3712
Joined: Sun Aug 10, 2008 1:15 pm
Location: Philippines

Re: What's the fastest time control you can effectively test

Post by Ferdy » Sat May 30, 2015 2:05 pm

jordanbray wrote:I've been running more tests to tune my engine, and have now gotten in the habbit of playing lots of games with certain features disabled, to see if anything was actually hurting the engine.

Turns out, at 40/.5+.01 time control, many of the evaluation features have performed poorly. However, I'm becoming a bit concerned that the time control is too fast.

However, running at this short of a time control enables me to run many more tests (46500 games overnight, with 31 (!) builds of the engine).

What time control do you most often test at, and what penalties exist for tuning to these fast time controls?
I usually use 60s + 100ms. Anything lower than that is just an experiment to see how values would perform. Candidate for release will use 120s + 100ms. There are times I use 90min +30s just to observe time allocation usage and ebf.
In my computer i7 2600k I don't get reliable result below 60s tc.

User avatar
Bloodbane
Posts: 154
Joined: Thu Oct 03, 2013 2:17 pm

Re: What's the fastest time control you can effectively test

Post by Bloodbane » Sat May 30, 2015 3:10 pm

I use 5s+0.05s and 15s+0.05s, and I could go as low as 1s+0.08s but anything lower than that starts causing time losses. I think these time controls are a bit fast and might have caused some quirks since Hakkapeliitta is strongest at short time controls (the shorter the better). However, this was a conscious choice which allows me to keep a fast development pace. Now that the amount of ideas I have is a lot smaller than before I am thinking of increasing the time controls.
Functional programming combines the flexibility and power of abstract mathematics with the intuitive clarity of abstract mathematics.
https://github.com/mAarnos

matthewlai
Posts: 787
Joined: Sun Aug 03, 2014 2:48 am
Location: London, UK
Contact:

Re: What's the fastest time control you can effectively test

Post by matthewlai » Sat May 30, 2015 6:40 pm

jordanbray wrote:I've been running more tests to tune my engine, and have now gotten in the habbit of playing lots of games with certain features disabled, to see if anything was actually hurting the engine.

Turns out, at 40/.5+.01 time control, many of the evaluation features have performed poorly. However, I'm becoming a bit concerned that the time control is too fast.

However, running at this short of a time control enables me to run many more tests (46500 games overnight, with 31 (!) builds of the engine).

What time control do you most often test at, and what penalties exist for tuning to these fast time controls?
I find that it's much better to play with low initial time and high increment, since at low increments, moves are made faster and faster towards the end, and long games become lightning games.

I use 1s + 0.5s, and games take about 30 seconds each. My engine can reliably run without time losses at 0.1s+0.01s using my own matching program (similar to cutechess), but most other engines I've tried couldn't.

In any case, I don't want to test that fast anyways, since real games aren't usually played that fast... and also because I have access to a cluster with hundreds of CPUs.

If you are not affiliated with a university (many of which have computing clusters), and don't mind spending some money, Amazon EC2 can rent you fairly fast Xeon cores for about $0.01 per core per hour. The trick is to use older generation spot instances that virtually don't fluctuate in price.
Disclosure: I work for DeepMind on the AlphaZero project, but everything I say here is personal opinion and does not reflect the views of DeepMind / Alphabet.

jordanbray
Posts: 51
Joined: Mon Aug 11, 2014 1:01 am

Re: What's the fastest time control you can effectively test

Post by jordanbray » Sat May 30, 2015 8:16 pm

Is there a program that can read in a pgn file (or, better, multiple pgn files) and report the elo difference? Something similar to cutechess's reporting?

User avatar
hgm
Posts: 23000
Joined: Fri Mar 10, 2006 9:06 am
Location: Amsterdam
Full name: H G Muller
Contact:

Re: What's the fastest time control you can effectively test

Post by hgm » Sat May 30, 2015 8:24 pm

I always uses BayesElo for that.

Adam Hair
Posts: 3199
Joined: Wed May 06, 2009 8:31 pm
Location: Fuquay-Varina, North Carolina

Re: What's the fastest time control you can effectively test

Post by Adam Hair » Sat May 30, 2015 8:46 pm

hgm wrote:I always uses BayesElo for that.
This is where I am suppose to extoll the virtues of Ordo :wink:
:lol:

jdart
Posts: 3676
Joined: Fri Mar 10, 2006 4:23 am
Location: http://www.arasanchess.org

Re: What's the fastest time control you can effectively test

Post by jdart » Sat May 30, 2015 11:39 pm

I am currently using 0:04+0.1 (4 sec + .1 sec increment). At this rate I am getting 10-11 ply searches typically, which seems to be adequate. And most engines do not lose on time at this TC.

--Jon

User avatar
lucasart
Posts: 3023
Joined: Mon May 31, 2010 11:29 am
Full name: lucasart
Contact:

Re: What's the fastest time control you can effectively test

Post by lucasart » Sun May 31, 2015 3:27 am

jordanbray wrote:I've been running more tests to tune my engine, and have now gotten in the habbit of playing lots of games with certain features disabled, to see if anything was actually hurting the engine.

Turns out, at 40/.5+.01 time control, many of the evaluation features have performed poorly. However, I'm becoming a bit concerned that the time control is too fast.

However, running at this short of a time control enables me to run many more tests (46500 games overnight, with 31 (!) builds of the engine).

What time control do you most often test at, and what penalties exist for tuning to these fast time controls?
Depends on your engine, and on your testing environment. Beware of overheads (read: Windows + bloated chess GUI). If you compile cutechess-cli from source (latest code from gituhb), you get a sub-millisecond timer (QElapsedTimer). Using Linux and cutechess-cli, I can test Stockfish patches in 1"+0.01", even using HT cores (ie. 7 concurrent games on 4 physical cores, 8 HT cores).

Problem is, sometimes results from extremely fast tc are not always (though often) good predictors of long tc results. That's very annoying, because there's no way you can know in advance (ie. without testing at short TC and then long TC).

Things to watch out for (read: test at short TC, then verify at long TC):
* king safety related
* passed pawn related
* search related (especially if kicks in at high depths, obviously)

Other eval patches are generally fine to test at hyper fast tc, and that's even beneficial, because it increases the sensitivity of the measure.

Another possibility for eval patches is to test at fixed depth (obviously not suitable for search patches). It works more than you can imagine. Given a choice between extreme tc (eg. 1+0.01) or fixed depth, fixed depth behaves better and should be prefered (read: higher draw rate for given throughput).
Theory and practice sometimes clash. And when that happens, theory loses. Every single time.

Post Reply