Debate: testing at fast time controls

Discussion of anything and everything relating to chess playing software and machines.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
User avatar
Kempelen
Posts: 620
Joined: Fri Feb 08, 2008 9:44 am
Location: Madrid - Spain
Contact:

Debate: testing at fast time controls

Post by Kempelen » Mon Dec 15, 2008 10:59 pm

For testing purposes where the majority of us test our engine in fast controls like 1m-1s or 1m-0s, do you think that if the engine improve in that fast control it will improve in slow/normal games? is there a correlation?

If not, what do you (assuming you test in fast games) do to take care of normal games?
Fermin Serrano
Author of 'Rodin' engine
http://sites.google.com/site/clonfsp/

BubbaTough
Posts: 1154
Joined: Fri Jun 23, 2006 3:18 am

Re: Debate: testing at fast time controls

Post by BubbaTough » Mon Dec 15, 2008 11:35 pm

Kempelen wrote:For testing purposes where the majority of us test our engine in fast controls like 1m-1s or 1m-0s, do you think that if the engine improve in that fast control it will improve in slow/normal games? is there a correlation?

If not, what do you (assuming you test in fast games) do to take care of normal games?
There is clearly a correlation, and equally clearly not a perfect correlation. The amusing thing is, if we tested with game in 30, due to advances in hardware, in 5 years it would be as if we tested in fast games anyway.

-Sam

bob
Posts: 20914
Joined: Mon Feb 27, 2006 6:30 pm
Location: Birmingham, AL

Re: Debate: testing at fast time controls

Post by bob » Tue Dec 16, 2008 12:51 am

Kempelen wrote:For testing purposes where the majority of us test our engine in fast controls like 1m-1s or 1m-0s, do you think that if the engine improve in that fast control it will improve in slow/normal games? is there a correlation?

If not, what do you (assuming you test in fast games) do to take care of normal games?
You probably won't like my answer, but the actual, factual answer is "nobody knows". I have tested changes at fast time controls and found improvement, then tested at longer time controls and found (a) similar improvement; (b) better improvement; (c) no improvement; (d) program actually played worse.

Based on many millions of games I have played, for the most part, changes that help at fast time controls also help at longer time controls. But the key words are "for the most part". Not "always or nearly always".

This is a real challenge...

User avatar
hgm
Posts: 24619
Joined: Fri Mar 10, 2006 9:06 am
Location: Amsterdam
Full name: H G Muller
Contact:

Re: Debate: testing at fast time controls

Post by hgm » Tue Dec 16, 2008 8:01 am

I think one of the problems is that techniques like null-move pruning lead to more and more 'unnatural' positions as the depth increases. Preferentially the engines try to refute everything through null moves, which means that the side playing the cut-nodes gets too lazy to cash in on any winning captures he can do, while the side playing the all-nodes plays nonsense most of the time, and in most branches simply does not get to cashing in on good captures. So the number of good captures on both sides tends to increase with depth, and this puts a heavier and heavier butdon on the implementation of QS. In very fast games an inefficient QS probably hurts much less than at long TC, where you really have to select carefully which one of a variety of captures you are going to try first.

Karmazen & Oliver
Posts: 374
Joined: Fri Mar 09, 2007 11:34 pm

Re: Debate: testing at fast time controls

Post by Karmazen & Oliver » Tue Dec 16, 2008 1:36 pm

Kempelen wrote:For testing purposes where the majority of us test our engine in fast controls like 1m-1s or 1m-0s, do you think that if the engine improve in that fast control it will improve in slow/normal games? is there a correlation?

If not, what do you (assuming you test in fast games) do to take care of normal games?
I think that is a conceptual error... I don´t think that a FAST results for a engine give good results versus Other engines in long time.s

bye. oliver from spain.

Marc Lacrosse
Posts: 511
Joined: Wed Mar 08, 2006 9:05 pm
Contact:

Re: Debate: testing at fast time controls

Post by Marc Lacrosse » Tue Dec 16, 2008 10:29 pm

Karmazen & Oliver wrote:
I think that is a conceptual error... I don´t think that a FAST results for a engine give good results versus Other engines in long time.s

bye. oliver from spain.
What you say is anything but thinking.
What are your arguments, what are your proofs?

I published some of mines long long ago.

Here :http://users.skynet.be/mlcc/chessbazaar/mlmfl.html

Instead of simply telling what you think please show us some real data.

If you do not have any data please do not add to common verbal pollution.

Marc

User avatar
M ANSARI
Posts: 3436
Joined: Thu Mar 16, 2006 6:10 pm

Re: Debate: testing at fast time controls

Post by M ANSARI » Wed Dec 17, 2008 8:19 am

To be honest I think it depends on the engine. Some engine results scale up almost perfectly ... Rybka is a perfect example. I remember thinking how silly it was to think that 1 1 mean anything ... but I was proven wrong.

User avatar
Kirill Kryukov
Posts: 492
Joined: Sun Mar 19, 2006 3:12 am

Re: Debate: testing at fast time controls

Post by Kirill Kryukov » Wed Dec 17, 2008 9:11 am

Kempelen wrote:For testing purposes where the majority of us test our engine in fast controls like 1m-1s or 1m-0s, do you think that if the engine improve in that fast control it will improve in slow/normal games? is there a correlation?

If not, what do you (assuming you test in fast games) do to take care of normal games?
I think that general assumption is that results of fast games correlate with the results of slow games very well. This is what we expect. I am yet to see any convincing data showing that it is not the case. I mean any example of engine that performs significantly different depending on time control. I think such example is necessary before this discussion can reach anywhere.

Best,
Kirill

Uri Blass
Posts: 8765
Joined: Wed Mar 08, 2006 11:37 pm
Location: Tel-Aviv Israel

Re: Debate: testing at fast time controls

Post by Uri Blass » Wed Dec 17, 2008 9:18 am

M ANSARI wrote:To be honest I think it depends on the engine. Some engine results scale up almost perfectly ... Rybka is a perfect example. I remember thinking how silly it was to think that 1 1 mean anything ... but I was proven wrong.
My opinion is that you do not learn to run before you know to walk.

It means that as long as the level of your program is weaker than rybka3 it is better to care only about blitz results.

If your program is better than rybka3 at blitz then it may be time to care about long time control.

Uri

F. Bluemers
Posts: 860
Joined: Thu Mar 09, 2006 10:21 pm
Location: Nederland
Contact:

Re: Debate: testing at fast time controls

Post by F. Bluemers » Wed Dec 17, 2008 7:14 pm

Kirill Kryukov wrote:
Kempelen wrote:For testing purposes where the majority of us test our engine in fast controls like 1m-1s or 1m-0s, do you think that if the engine improve in that fast control it will improve in slow/normal games? is there a correlation?

If not, what do you (assuming you test in fast games) do to take care of normal games?
I think that general assumption is that results of fast games correlate with the results of slow games very well. This is what we expect. I am yet to see any convincing data showing that it is not the case. I mean any example of engine that performs significantly different depending on time control. I think such example is necessary before this discussion can reach anywhere.

Best,
Kirill
What's it about is this:
Can we tune an engine with long time controls and asume it would do fine on short tc's as well and vice versa.
(edit: I mean,it might not be the same)
Best
Fonzy

Post Reply