For testing purposes where the majority of us test our engine in fast controls like 1m-1s or 1m-0s, do you think that if the engine improve in that fast control it will improve in slow/normal games? is there a correlation?
If not, what do you (assuming you test in fast games) do to take care of normal games?
Debate: testing at fast time controls
Moderators: hgm, Rebel, chrisw
-
- Posts: 620
- Joined: Fri Feb 08, 2008 10:44 am
- Location: Madrid - Spain
-
- Posts: 1154
- Joined: Fri Jun 23, 2006 5:18 am
Re: Debate: testing at fast time controls
There is clearly a correlation, and equally clearly not a perfect correlation. The amusing thing is, if we tested with game in 30, due to advances in hardware, in 5 years it would be as if we tested in fast games anyway.Kempelen wrote:For testing purposes where the majority of us test our engine in fast controls like 1m-1s or 1m-0s, do you think that if the engine improve in that fast control it will improve in slow/normal games? is there a correlation?
If not, what do you (assuming you test in fast games) do to take care of normal games?
-Sam
-
- Posts: 20943
- Joined: Mon Feb 27, 2006 7:30 pm
- Location: Birmingham, AL
Re: Debate: testing at fast time controls
You probably won't like my answer, but the actual, factual answer is "nobody knows". I have tested changes at fast time controls and found improvement, then tested at longer time controls and found (a) similar improvement; (b) better improvement; (c) no improvement; (d) program actually played worse.Kempelen wrote:For testing purposes where the majority of us test our engine in fast controls like 1m-1s or 1m-0s, do you think that if the engine improve in that fast control it will improve in slow/normal games? is there a correlation?
If not, what do you (assuming you test in fast games) do to take care of normal games?
Based on many millions of games I have played, for the most part, changes that help at fast time controls also help at longer time controls. But the key words are "for the most part". Not "always or nearly always".
This is a real challenge...
-
- Posts: 27808
- Joined: Fri Mar 10, 2006 10:06 am
- Location: Amsterdam
- Full name: H G Muller
Re: Debate: testing at fast time controls
I think one of the problems is that techniques like null-move pruning lead to more and more 'unnatural' positions as the depth increases. Preferentially the engines try to refute everything through null moves, which means that the side playing the cut-nodes gets too lazy to cash in on any winning captures he can do, while the side playing the all-nodes plays nonsense most of the time, and in most branches simply does not get to cashing in on good captures. So the number of good captures on both sides tends to increase with depth, and this puts a heavier and heavier butdon on the implementation of QS. In very fast games an inefficient QS probably hurts much less than at long TC, where you really have to select carefully which one of a variety of captures you are going to try first.
-
- Posts: 374
- Joined: Sat Mar 10, 2007 12:34 am
Re: Debate: testing at fast time controls
I think that is a conceptual error... I don´t think that a FAST results for a engine give good results versus Other engines in long time.sKempelen wrote:For testing purposes where the majority of us test our engine in fast controls like 1m-1s or 1m-0s, do you think that if the engine improve in that fast control it will improve in slow/normal games? is there a correlation?
If not, what do you (assuming you test in fast games) do to take care of normal games?
bye. oliver from spain.
-
- Posts: 511
- Joined: Wed Mar 08, 2006 10:05 pm
Re: Debate: testing at fast time controls
What you say is anything but thinking.Karmazen & Oliver wrote:
I think that is a conceptual error... I don´t think that a FAST results for a engine give good results versus Other engines in long time.s
bye. oliver from spain.
What are your arguments, what are your proofs?
I published some of mines long long ago.
Here :http://users.skynet.be/mlcc/chessbazaar/mlmfl.html
Instead of simply telling what you think please show us some real data.
If you do not have any data please do not add to common verbal pollution.
Marc
-
- Posts: 3707
- Joined: Thu Mar 16, 2006 7:10 pm
Re: Debate: testing at fast time controls
To be honest I think it depends on the engine. Some engine results scale up almost perfectly ... Rybka is a perfect example. I remember thinking how silly it was to think that 1 1 mean anything ... but I was proven wrong.
-
- Posts: 492
- Joined: Sun Mar 19, 2006 4:12 am
Re: Debate: testing at fast time controls
I think that general assumption is that results of fast games correlate with the results of slow games very well. This is what we expect. I am yet to see any convincing data showing that it is not the case. I mean any example of engine that performs significantly different depending on time control. I think such example is necessary before this discussion can reach anywhere.Kempelen wrote:For testing purposes where the majority of us test our engine in fast controls like 1m-1s or 1m-0s, do you think that if the engine improve in that fast control it will improve in slow/normal games? is there a correlation?
If not, what do you (assuming you test in fast games) do to take care of normal games?
Best,
Kirill
-
- Posts: 10297
- Joined: Thu Mar 09, 2006 12:37 am
- Location: Tel-Aviv Israel
Re: Debate: testing at fast time controls
My opinion is that you do not learn to run before you know to walk.M ANSARI wrote:To be honest I think it depends on the engine. Some engine results scale up almost perfectly ... Rybka is a perfect example. I remember thinking how silly it was to think that 1 1 mean anything ... but I was proven wrong.
It means that as long as the level of your program is weaker than rybka3 it is better to care only about blitz results.
If your program is better than rybka3 at blitz then it may be time to care about long time control.
Uri
-
- Posts: 868
- Joined: Thu Mar 09, 2006 11:21 pm
- Location: Nederland
Re: Debate: testing at fast time controls
What's it about is this:Kirill Kryukov wrote:I think that general assumption is that results of fast games correlate with the results of slow games very well. This is what we expect. I am yet to see any convincing data showing that it is not the case. I mean any example of engine that performs significantly different depending on time control. I think such example is necessary before this discussion can reach anywhere.Kempelen wrote:For testing purposes where the majority of us test our engine in fast controls like 1m-1s or 1m-0s, do you think that if the engine improve in that fast control it will improve in slow/normal games? is there a correlation?
If not, what do you (assuming you test in fast games) do to take care of normal games?
Best,
Kirill
Can we tune an engine with long time controls and asume it would do fine on short tc's as well and vice versa.
(edit: I mean,it might not be the same)
Best
Fonzy