Elo Increase per Doubling

[petero2]

@Don

What is the value of the asymptote to your red curve? (This is of course the maximal Elo in your system).

Is that a joke? I wish I knew where it all ended

I think if we gathered this data for several more levels and got enough data for each point we could try to estimate the value using some type of curve fitting.

If someone wants to take a crack at estimating this with curve fitting, here is the final result:

Code: Select all

Rank Name    Elo      +      -    games   score   oppo.   draws 
   1 10    3127.9   25.4   25.4    2739   94.2%  2328.9   10.4% 
   2 09    2997.0   21.6   21.6    2739   86.8%  2341.9   15.2% 
   3 08    2891.6   20.4   20.4    2740   79.9%  2352.8   15.5% 
   4 07    2752.0   19.9   19.9    2740   70.2%  2366.7   15.8% 
   5 06    2615.3   20.2   20.2    2740   60.8%  2380.4   14.5% 
   6 05    2467.2   20.9   20.9    2740   51.2%  2395.2   13.0% 
   7 04    2307.4   22.1   22.1    2740   41.4%  2411.2   10.5% 
   8 03    2118.3   23.5   23.5    2740   30.6%  2430.1    8.6% 
   9 02    1932.7   25.3   25.3    2740   20.9%  2448.6    7.8% 
  10 01    1709.7   28.6   28.6    2740   10.9%  2470.9    5.3% 
  11 00    1500.0   36.9   36.9    2740    3.2%  2491.9    3.1% 

After some trial and error: Elo=1500+2350*ln(1+Name/10)
.

[Adam Hair]

Thanks a lot, Adam! Exactly the data I was searching for!

I made a plot graph out of it:


(With Delta Elo(x) = Elo(x)-Elo(x-1) )

So there IS a diminishing return, when depth increases! Actually, data from plies 1-5 would be very interesting as well!

Here is an estimated best fit for the Fruit results (I limited the estimate to two parameters):

Elo Delta = 551*Depth^(-0.724)




And for Houdini (after throwing out the first Elo Delta value due to it being an outlier):

Elo Delta = 681*Depth^(-0.903)

[Adam Hair]

Adam, what are you using for 2^n*time versus 1*time used at short controls? I played a bit with fixed ply games for n+1 versus n plies some half a year ago, my impression is that the unrelated engines have pretty unique behaviour with respect to increased depth, I thought of inventing "mostly search similarity tool" (in contrast to mostly eval Sim03 tester, if I understood something) based on the shape of the Elo increase.

Here is the picture for Houdini 1.5 (if I found the correct file)



For StockFish 2.1




I had some results for IvanHoe, which was closer in its shape to Houdini than to StockFish.


The Branching Factor of Houdini 1.5 with respect to depth:



So, it seems that the delta Elo with the depth decreases (with some bumping odd-even plies), and with respect with the number of doublings in time decreases even further, because BF slowly increases.

Kai

For the short time control, the base time was 6 seconds + 0.1 seconds per move.

[Don]

@Don

What is the value of the asymptote to your red curve? (This is of course the maximal Elo in your system).

Is that a joke? I wish I knew where it all ended

I think if we gathered this data for several more levels and got enough data for each point we could try to estimate the value using some type of curve fitting.

If someone wants to take a crack at estimating this with curve fitting, here is the final result:

Code: Select all

Rank Name    Elo      +      -    games   score   oppo.   draws 
   1 10    3127.9   25.4   25.4    2739   94.2%  2328.9   10.4% 
   2 09    2997.0   21.6   21.6    2739   86.8%  2341.9   15.2% 
   3 08    2891.6   20.4   20.4    2740   79.9%  2352.8   15.5% 
   4 07    2752.0   19.9   19.9    2740   70.2%  2366.7   15.8% 
   5 06    2615.3   20.2   20.2    2740   60.8%  2380.4   14.5% 
   6 05    2467.2   20.9   20.9    2740   51.2%  2395.2   13.0% 
   7 04    2307.4   22.1   22.1    2740   41.4%  2411.2   10.5% 
   8 03    2118.3   23.5   23.5    2740   30.6%  2430.1    8.6% 
   9 02    1932.7   25.3   25.3    2740   20.9%  2448.6    7.8% 
  10 01    1709.7   28.6   28.6    2740   10.9%  2470.9    5.3% 
  11 00    1500.0   36.9   36.9    2740    3.2%  2491.9    3.1% 

After some trial and error: Elo=1500+2350*ln(1+Name/10)
.

It seems to be a really good fit, but I extrapolated it to level 1000 and it shows an ELO of over 12000!

If I go to 100000 it comes out to almost 18,000 ELO!

I don't believe perfect play goes that high. If I'm not mistaken you can achieve an arbitrarily high ELO with this formula - there is no asymptote so I guess it's just a good approximation at these low levels.

Do you have any other models that might impose some sort of limit?

[petero2]

Yes, the logarithmic formula goes to infinity, so not a good approximation for large x.

Here is an exponential fit instead: Elo=1495+2996*(1-exp(-Name/12.8))


And extrapolated:

[JuLieN]

Yes, the logarithmic formula goes to infinity, so not a good approximation for large x.

And extrapolated:

Tack Peter, that's more like it and it fits our intuition.

[Adam Hair]

@Don

What is the value of the asymptote to your red curve? (This is of course the maximal Elo in your system).

Is that a joke? I wish I knew where it all ended

I think if we gathered this data for several more levels and got enough data for each point we could try to estimate the value using some type of curve fitting.

If someone wants to take a crack at estimating this with curve fitting, here is the final result:

Code: Select all

Rank Name    Elo      +      -    games   score   oppo.   draws 
   1 10    3127.9   25.4   25.4    2739   94.2%  2328.9   10.4% 
   2 09    2997.0   21.6   21.6    2739   86.8%  2341.9   15.2% 
   3 08    2891.6   20.4   20.4    2740   79.9%  2352.8   15.5% 
   4 07    2752.0   19.9   19.9    2740   70.2%  2366.7   15.8% 
   5 06    2615.3   20.2   20.2    2740   60.8%  2380.4   14.5% 
   6 05    2467.2   20.9   20.9    2740   51.2%  2395.2   13.0% 
   7 04    2307.4   22.1   22.1    2740   41.4%  2411.2   10.5% 
   8 03    2118.3   23.5   23.5    2740   30.6%  2430.1    8.6% 
   9 02    1932.7   25.3   25.3    2740   20.9%  2448.6    7.8% 
  10 01    1709.7   28.6   28.6    2740   10.9%  2470.9    5.3% 
  11 00    1500.0   36.9   36.9    2740    3.2%  2491.9    3.1% 

After some trial and error: Elo=1500+2350*ln(1+Name/10)
.

It seems to be a really good fit, but I extrapolated it to level 1000 and it shows an ELO of over 12000!

If I go to 100000 it comes out to almost 18,000 ELO!

I don't believe perfect play goes that high. If I'm not mistaken you can achieve an arbitrarily high ELO with this formula - there is no asymptote so I guess it's just a good approximation at these low levels.

Do you have any other models that might impose some sort of limit?

Elo = 6028.21*e^(-12.48/(Name+8.95))

This equation fits your data well and is asymtopic to Elo = 6028.21.

Of course, 6028.21 Elo is only relevant to the Elo values assigned in your data. But, your engine, limited to only 2^18 nodes per move, seems to be ~2900 Elo away from perfection. If you let it search 2^1,000,000 nodes per move, Komodo will play virtually perfect chess (according to this model, its Elo would be 6028.13). 2^10,000 nodes/move would be within 8 Elo of perfection.

[Don]

Yes, the logarithmic formula goes to infinity, so not a good approximation for large x.

Here is an exponential fit instead: Elo=1495+2996*(1-exp(-Name/12.8))


And extrapolated:

Awesome! So now we know God's ELO! It's about 4491!

Actually, I expect perfect play to be well over 4000 ELO - this is pretty believable but of course I don't take it too seriously. Maybe I should test a few more levels with a few more games and fiddle with the constants and see what happens!

[Don]

@Don

What is the value of the asymptote to your red curve? (This is of course the maximal Elo in your system).

Is that a joke? I wish I knew where it all ended

I think if we gathered this data for several more levels and got enough data for each point we could try to estimate the value using some type of curve fitting.

If someone wants to take a crack at estimating this with curve fitting, here is the final result:

Code: Select all

Rank Name    Elo      +      -    games   score   oppo.   draws 
   1 10    3127.9   25.4   25.4    2739   94.2%  2328.9   10.4% 
   2 09    2997.0   21.6   21.6    2739   86.8%  2341.9   15.2% 
   3 08    2891.6   20.4   20.4    2740   79.9%  2352.8   15.5% 
   4 07    2752.0   19.9   19.9    2740   70.2%  2366.7   15.8% 
   5 06    2615.3   20.2   20.2    2740   60.8%  2380.4   14.5% 
   6 05    2467.2   20.9   20.9    2740   51.2%  2395.2   13.0% 
   7 04    2307.4   22.1   22.1    2740   41.4%  2411.2   10.5% 
   8 03    2118.3   23.5   23.5    2740   30.6%  2430.1    8.6% 
   9 02    1932.7   25.3   25.3    2740   20.9%  2448.6    7.8% 
  10 01    1709.7   28.6   28.6    2740   10.9%  2470.9    5.3% 
  11 00    1500.0   36.9   36.9    2740    3.2%  2491.9    3.1% 

After some trial and error: Elo=1500+2350*ln(1+Name/10)
.

It seems to be a really good fit, but I extrapolated it to level 1000 and it shows an ELO of over 12000!

If I go to 100000 it comes out to almost 18,000 ELO!

I don't believe perfect play goes that high. If I'm not mistaken you can achieve an arbitrarily high ELO with this formula - there is no asymptote so I guess it's just a good approximation at these low levels.

Do you have any other models that might impose some sort of limit?

Elo = 6028.21*e^(-12.48/(Name+8.95))

This equation fits your data well and is asymtopic to Elo = 6028.21.

Of course, 6028.21 Elo is only relevant to the Elo values assigned in your data. But, your engine, limited to only 2^18 nodes per move, seems to be ~2900 Elo away from perfection. If you let it search 2^1,000,000 nodes per move, Komodo will play virtually perfect chess (according to this model, its Elo would be 6028.13). 2^10,000 nodes/move would be within 8 Elo of perfection.

I find 4500 to be a much more believable figure than 6028 - but what do I know? It could be anything!

BTW I adjusted my data so that version 00 is calibrated to 1500 ELO - I think that is within about 100 or 200 ELO of the truth. Komodo is doing about 3.18 ply on average. At level 02, Komodo would be about 433 ELO stronger or 1933 doing on average 5.37 ply.

What would be interesting is to run the same experiment with a different program using the same basic formula (with adjusted constants) and see if they agree!

[Laskos]

For the short time control, the base time was 6 seconds + 0.1 seconds per move.

I mean, what interface are you using to assign assymmetrical time controls? LittleBlitzer can't do this. Or, how Don assigns fixed, but different nodes and depths? I managed to get fixed, different depths only in Fritz and Aquarium.

Kai