Elo points gain from doubling time

[Ajedrecista]

Hello:

I do not know if my post is off-topic or not, but Don posted an experiment some months ago:

Computer Chess Scalability Study

Here is the thread where Don exposed his experiment. There are also more studies in the same issue with the participation of Peter Österlund and Adam Hair:

Elo versus speed

Elo Increase per Doubling

I hope these links will be useful for comparison.

Regards from Spain.

Ajedrecista.

[Laskos]

Hello:

I do not know if my post is off-topic or not, but Don posted an experiment some months ago:

Computer Chess Scalability Study

Here is the thread where Don exposed his experiment. There are also more studies in the same issue with the participation of Peter Österlund and Adam Hair:

Elo versus speed

Elo Increase per Doubling

I hope these links will be useful for comparison.

Regards from Spain.

Ajedrecista.

Thanks, so most of it was been done before, although my time controls are in the upper limits of other tests. The results seem consistent with mine.

[Rebel]

Hello:

I do not know if my post is off-topic or not, but Don posted an experiment some months ago:

Computer Chess Scalability Study

Here is the thread where Don exposed his experiment. There are also more studies in the same issue with the participation of Peter Österlund and Adam Hair:

Elo versus speed

Elo Increase per Doubling

I hope these links will be useful for comparison.

Regards from Spain.

Ajedrecista.

Thanks, so most of it was been done before, although my time controls are in the upper limits of other tests. The results seem consistent with mine.

Here is another study about the diminishing return effect.

http://www.top-5000.nl/ply.htm

[Laskos]

These 2,000-4,000 blitz games matches take a lot of time, so I will not throw in different TC Houdinis, as I just wanted to see the rule of thumb law.

Well, the 'rule of thumb' seems to be 100 Elo per doubling. (Which, btw, is more than I expected. I always assumed 70 Elo per doubling.) The rest of the analysis seems mostly analysing noise, based on a +7 +/- 4.2 @1s and a -19 +/- 6.3 @ 4s.

A bit off the noise, besides that, if I am lucky, I could fall inside 1SD error margins. Also, 2SD fit still shows a decrease with time in gain (but large uncertainties). I saw other studies shown to me by Jesus Munoz, and it seems that the rule of thumb 107*(time in seconds per move)^(-0.20) fits the data of others pretty well. Yes, the errors are large, but playing more than 2,000 games at 4s/move TC is a pain. I focused specifically at these longer TC, because at ultra-short controls it's easy to see 150-200 points per doubling, but it's irrelevant.

[Don]

The result is weird anyway: the draw fraction seems to go up enormously at higher depth. I would consider 80% draws a ridiculously high draw fraction, between nearly equal engines.

I don't think it is - the number of draws goes up pretty steadily with the quality of the players. I think in a few years we are going to see the ratings of the top programs get really compressed due to this. You will play a 100 games match and they will be mostly draws. It will be like checkers is now.

[Laskos]

The result is weird anyway: the draw fraction seems to go up enormously at higher depth. I would consider 80% draws a ridiculously high draw fraction, between nearly equal engines.

I don't think it is - the number of draws goes up pretty steadily with the quality of the players. I think in a few years we are going to see the ratings of the top programs get really compressed due to this. You will play a 100 games match and they will be mostly draws. It will be like checkers is now.

It was Houdini 3 ply 14 vs ply 15, ultra-bullet, 80% draws are impossible. The highest I seen at very long time controls is 73% or so.

[Don]

These 2,000-4,000 blitz games matches take a lot of time, so I will not throw in different TC Houdinis, as I just wanted to see the rule of thumb law.

Well, the 'rule of thumb' seems to be 100 Elo per doubling. (Which, btw, is more than I expected. I always assumed 70 Elo per doubling.) The rest of the analysis seems mostly analysing noise, based on a +7 +/- 4.2 @1s and a -19 +/- 6.3 @ 4s.

The elo value per doubling drops off rather sharply with each additional doubling so I don't think there is simple rule of thumb. I have done many experiments that show this. Again, I think it's primarily due the increased number of draws as they tend to go up very steadily with depth too.

It would be interesting to study that too. Imagine doing one of these logarithmic studies where you throw out all draws. You might have a much flatter rating curve.

[Don]

The result is weird anyway: the draw fraction seems to go up enormously at higher depth. I would consider 80% draws a ridiculously high draw fraction, between nearly equal engines.

I don't think it is - the number of draws goes up pretty steadily with the quality of the players. I think in a few years we are going to see the ratings of the top programs get really compressed due to this. You will play a 100 games match and they will be mostly draws. It will be like checkers is now.

It was Houdini 3 ply 14 vs ply 15, ultra-bullet, 80% draws are impossible. The highest I seen at very long time controls is 73% or so.

I should have qualified my statement - it seems ridiculously high at that time control for sure.

I'm running a 30 second test right now to test something that looks good at much faster time controls and I'm getting this:

Code: Select all

white wins 908
black wins 628
64.67 percent draws 
4,347 games

It shows that white does indeed have an advantage and even at this relatively fast time control almost 65% of the games are draws.

I think in the not so distant future you will see very few decisive games and almost all of them will be by white.

I happened to have some games played at 0.8 seconds per games + 0.008 increment and I get this:

Code: Select all

white wins 12913
black wins 11499
36.30 percent draws
about 38,000 games

You can see the trend, with increasing strength the draw ratio rises non-trivially and of the remaining games white tends to wins a higher percentage. This is far from comprehensive of course but I have consistently seen it - at least with Komodo.

Another problem I have identified is the contempt factor. I am of the opinion that the relative strength of the opponent should be communicated to the engines and built it to the protocol because this is becoming an important issue too. It can be worth 100 ELO or more if you are playing way up or down by several hundred ELO (I studied this too.) Unless that is handled a perfect player will draw far too many games against weak opponents. In human play it's rare not to have a rough idea of the strength of your opponent. That can be communicated via some user defined contempt factor but I would like to see it built in to the protocol.

Don

[Houdini]

It was Houdini 3 ply 14 vs ply 15, ultra-bullet, 80% draws are impossible. The highest I seen at very long time controls is 73% or so.

Correct, with Houdini 3 this is nearly impossible - it suggests that the opening positions for the test are not well chosen.
Even at long time control Houdini will have close to 50% decided games. See for example the 90 min+30 sec/move tests I played with the Houdini 3 beta against Houdini 2, Stockfish 2.3.1 and Komodo 5, which over-all was +135 -50 =175.

Robert

[Uri Blass]

It was Houdini 3 ply 14 vs ply 15, ultra-bullet, 80% draws are impossible. The highest I seen at very long time controls is 73% or so.

Correct, with Houdini 3 this is nearly impossible - it suggests that the opening positions for the test are not well chosen.
Even at long time control Houdini will have close to 50% decided games. See for example the 90 min+30 sec/move tests I played with the Houdini 3 beta against Houdini 2, Stockfish 2.3.1 and Komodo 5, which over-all was +135 -50 =175.

Robert

Note that it is possible that having houdini3 against houdini3 and not against other programs increase the number of draws.

I still think that 15 plies against 14 plies should not give so many draws unless you have a bad choice of the opening positions(for example using a big book that cause the engines to start to play only at move 20 or move 30 is a bad idea)