Doubling of time control

[corres]

Is there any information about the participants of the test matches and the opening book used?
Without these infos the data set published is hanging in the air.
Basically I think that the Elo system has no fundamental issue but the power of a chess engine really and relatively downfalls with the enhancement of move time.
This can be attributed to the pruning and the fact that the number of the possible variations grows with moving time.

[Guenther]

Is there any information about the participants of the test matches and the opening book used?
Without these infos the data set published is hanging in the air.
...

What test matches do you mean? The original thread and provided data
is about Komodo 9.3 and a set of 1500 openings (this info is in the first post).

[fastgm]

Hello Robert,

perhaps the conditions in my initial post wasn't detailed enough.

Openings:
1500 different opening positions with changing colors = 3000 games for each match

Engine:
Komodo 9.3 against itself

Andreas

[fastgm]

Thanks Kai, i have asked Ferdinand if he can help me with such a tool.

[fastgm]

Ferdinand Mosca helped me. Thanks a lot!

He found a program (Protools) written by Ed Schröder:
http://www.top-5000.nl/dl/protools15.zip

and the latest prodeo.exe
http://www.talkchess.com/forum/viewtopi ... 77&t=61721

With this tool I have created the following additional data:

Code: Select all

Engine                    Depth       Time   Games     Moves  Average Forfeit Book
Komodo 9.3 T1 20+0.2      17.40   28:34:57    3000    256891    0.40     0   26737 (8.91)
Komodo 9.3 T1 10+0.1      15.44   14:19:45    3000    256342    0.20     0   24694 (8.23)

Engine                    Depth       Time   Games     Moves  Average Forfeit Book
Komodo 9.3 T1 40+0.4      19.05   58:18:17    3000    265755    0.79     0   27193 (9.06)
Komodo 9.3 T1 20+0.2      17.07   29:24:34    3000    265279    0.40     0   25454 (8.48)

Engine                    Depth       Time   Games     Moves  Average Forfeit Book
Komodo 9.3 T1 80+0.8      20.70  116:39:50    3000    267468    1.57     0   27456 (9.15)
Komodo 9.3 T1 40+0.4      18.85   58:53:16    3000    267048    0.79     0   25729 (8.58)

Engine                    Depth       Time   Games     Moves  Average Forfeit Book
Komodo 9.3 T1 160+1.6     22.45  234:51:52    3000    267920    3.16     0   27371 (9.12)
Komodo 9.3 T1 80+0.8      20.50  118:06:24    3000    267555    1.59     0   26095 (8.70)

Engine                    Depth       Time   Games     Moves  Average Forfeit Book
Komodo 9.3 T1 320+3.2     24.30  476:30:53    3000    274164    6.26     0   27771 (9.26)
Komodo 9.3 T1 160+1.6     22.29  239:40:46    3000    273826    3.15     0   26428 (8.81)

Engine                    Depth       Time   Games     Moves  Average Forfeit Book
Komodo 9.3 T1 640+6.4     26.28  950:20:50    3000    272343    12.56    0   27972 (9.32)
Komodo 9.3 T1 320+3.2     24.26  478:20:21    3000    272091    6.33     0   26638 (8.88)

Engine                    Depth        Time  Games     Moves  Average Forfeit Book
Komodo 9.3 T1 1280+12.8   28.09  1908:54:31   3000    276907    24.82    0   28475 (9.49)
Komodo 9.3 T1 640+6.4     26.18   960:08:07   3000    276750    12.49    0   27004 (9.00)

Engine                    Depth        Time  Games     Moves  Average Forfeit Book
Komodo 9.3 T1 2560+25.6   29.92  3806:05:02   3000    275195    49.79    0   28760 (9.59)
Komodo 9.3 T1 1280+12.8   28.01  1914:36:07   3000    275034    25.06    0   27544 (9.18)


Time control comparison between engines

Depth     : Average search depth
Time      : Total time engine used
Moves     : Total moves engine played
Average   : Average time per move in centi-seconds
Forfeit   : Games engine lost due to time forfeit

List is sorted on Average Time indicating the engine that uses the most time tops.

[Guenther]

Ferdinand Mosca helped me. Thanks a lot!

He found a program (Protools) written by Ed Schröder:
http://www.top-5000.nl/dl/protools15.zip

and the latest prodeo.exe
http://www.talkchess.com/forum/viewtopi ... 77&t=61721

With this tool I have created the following additional data:

Code: Select all

Engine                    Depth       Time   Games     Moves  Average Forfeit Book
Komodo 9.3 T1 20+0.2      17.40   28:34:57    3000    256891    0.40     0   26737 (8.91)
Komodo 9.3 T1 10+0.1      15.44   14:19:45    3000    256342    0.20     0   24694 (8.23)

Engine                    Depth       Time   Games     Moves  Average Forfeit Book
Komodo 9.3 T1 40+0.4      19.05   58:18:17    3000    265755    0.79     0   27193 (9.06)
Komodo 9.3 T1 20+0.2      17.07   29:24:34    3000    265279    0.40     0   25454 (8.48)

Engine                    Depth       Time   Games     Moves  Average Forfeit Book
Komodo 9.3 T1 80+0.8      20.70  116:39:50    3000    267468    1.57     0   27456 (9.15)
Komodo 9.3 T1 40+0.4      18.85   58:53:16    3000    267048    0.79     0   25729 (8.58)

Engine                    Depth       Time   Games     Moves  Average Forfeit Book
Komodo 9.3 T1 160+1.6     22.45  234:51:52    3000    267920    3.16     0   27371 (9.12)
Komodo 9.3 T1 80+0.8      20.50  118:06:24    3000    267555    1.59     0   26095 (8.70)

Engine                    Depth       Time   Games     Moves  Average Forfeit Book
Komodo 9.3 T1 320+3.2     24.30  476:30:53    3000    274164    6.26     0   27771 (9.26)
Komodo 9.3 T1 160+1.6     22.29  239:40:46    3000    273826    3.15     0   26428 (8.81)

Engine                    Depth       Time   Games     Moves  Average Forfeit Book
Komodo 9.3 T1 640+6.4     26.28  950:20:50    3000    272343    12.56    0   27972 (9.32)
Komodo 9.3 T1 320+3.2     24.26  478:20:21    3000    272091    6.33     0   26638 (8.88)

Engine                    Depth        Time  Games     Moves  Average Forfeit Book
Komodo 9.3 T1 1280+12.8   28.09  1908:54:31   3000    276907    24.82    0   28475 (9.49)
Komodo 9.3 T1 640+6.4     26.18   960:08:07   3000    276750    12.49    0   27004 (9.00)

Engine                    Depth        Time  Games     Moves  Average Forfeit Book
Komodo 9.3 T1 2560+25.6   29.92  3806:05:02   3000    275195    49.79    0   28760 (9.59)
Komodo 9.3 T1 1280+12.8   28.01  1914:36:07   3000    275034    25.06    0   27544 (9.18)


Time control comparison between engines

Depth     : Average search depth
Time      : Total time engine used
Moves     : Total moves engine played
Average   : Average time per move in centi-seconds
Forfeit   : Games engine lost due to time forfeit

List is sorted on Average Time indicating the engine that uses the most time tops.

This is great! I will look closer on it next week and also try to make updated graphs, but probably Kai will be faster ;-)

Note also that we even have 6000 games for an average depth for all entries except the first and the last ones.

Code: Select all

Engine                  Depth   Time            Games   Moves   Average Forfeit Book           Avg.D   G       Avg.D Inc
Komodo 9.3 T1 10+0.1    15,44   14:19:45        3000    256342  0,20    0       24694   8,23   15,440  3000    
Komodo 9.3 T1 20+0.2    17,07   29:24:34        3000    265279  0,40    0       25454   8,48   17,235  6000    1,795
Komodo 9.3 T1 20+0.2    17,40   28:34:57        3000    256891  0,40    0       26737   8,91                   
Komodo 9.3 T1 40+0.4    18,85   58:53:16        3000    267048  0,79    0       25729   8,58   18,950  6000    1,715
Komodo 9.3 T1 40+0.4    19,05   58:18:17        3000    265755  0,79    0       27193   9,06                   
Komodo 9.3 T1 80+0.8    20,50   118:06:24       3000    267555  1,59    0       26095   8,70   20,600  6000    1,650
Komodo 9.3 T1 80+0.8    20,70   116:39:50       3000    267468  1,57    0       27456   9,15                   
Komodo 9.3 T1 160+1.6   22,29   239:40:46       3000    273826  3,15    0       26428   8,81   22,370  6000    1,770
Komodo 9.3 T1 160+1.6   22,45   234:51:52       3000    267920  3,16    0       27371   9,12                   
Komodo 9.3 T1 320+3.2   24,26   478:20:21       3000    272091  6,33    0       26638   8,88   24,280  6000    1,910
Komodo 9.3 T1 320+3.2   24,30   476:30:53       3000    274164  6,26    0       27771   9,26                   
Komodo 9.3 T1 640+6.4   26,18   960:08:07       3000    276750  12,49   0       27004   9,00   26,230  6000    1,950
Komodo 9.3 T1 640+6.4   26,28   950:20:50       3000    272343  12,56   0       27972   9,32                   
Komodo 9.3 T1 1280+12.8 28,01   1914:36:07      3000    275034  25,06   0       27544   9,18   28,050  6000    1,715
Komodo 9.3 T1 1280+12.8 28,09   1908:54:31      3000    276907  24,82   0       28475   9,49                   
Komodo 9.3 T1 2560+25.6 29,92   3806:05:02      3000    275195  49,79   0       28760   9,59   29,920  3000    1,870

[Guenther]

Hello Robert,

perhaps the conditions in my initial post wasn't detailed enough.

Openings:
1500 different opening positions with changing colors = 3000 games for each match

Engine:
Komodo 9.3 against itself

Andreas

BTW what resign and/or adjudication options were in charg

Guenther

[fastgm]

No adjudication (draw, resign) options.

[Ajedrecista]

Hello:

I have been looking for an adjust of this data and I think I have something decent. Here I go:

I looked into a Gompertz function (an example is here) and I came with the following:

1.- I converted accumulated Elo gain (0, 144, 277,...) into score with the Elo model µ = 1/[1 + 10^(-Elo/400)].

2.- I used the TC values of 0, 1, 2 and so on in the horizontal axis, like it is seen in the cited paper just after equation 1.

3.- I used the numbers ln[ln(µ_1/µ_0)], ln[ln(µ_2/µ_1)], ..., ln[ln(µ_8/µ_7)] in the vertical axis. (Equation 4 of the paper).

4.- I did a linear regression with Excel to obtain beta and gamma parameters:

Code: Select all

Gompertz fit:
Fitted_µ = alpha*exp[-beta*exp(-gamma*TC)]

Linear fit of the 8 data points = m*TC + n ~ -0.64087232*TC - 0.51556368 (R² ~ 0.99744438)

(Equation 4): gamma = -m ~ 0.64087232
(Equation 4): beta = exp(n)/[exp(gamma) - 1] ~ 0.66489741

5.- By definition, alpha is the saturation level, so we can expect that max(µ) = 1 = alpha --> horizontal asymptote. If that:

Code: Select all

Fitted_µ ~ exp[-0.66489741*exp(-gamma*0.64087232)]

Converting fitted_µ into Elo gain (rounding up to the nearest Elo integer):

TC  Elo   Fitted Elo   Elo - (fitted Elo)
 1  144       151               -7
 2  277       277                0
 3  389       396               -9
 4  490       512              -22
 5  583       625              -42
 6  656       738              -82
 7  715       850             -135
 8  766       961             -195

Average error = -61.5 Elo

6.- Equation 5 of the paper proposes the following:

Code: Select all

alpha_TC = exp[ln(µ_TC) + beta*exp(-gamma*TC)]

I obtain 8 values of alpha_TC. If I randomly choose alpha = average(alpha_TC) ~ 0.99310185

Fitted_µ ~ 0.99310185*exp[-0.66489741*exp(-gamma*0.64087232)]

TC  Elo   Fitted Elo   Elo - (fitted Elo)
 1  144       147               -3
 2  277       270               -7
 3  389       384               +5
 4  490       489               +1
 5  583       585               -2
 6  656       668              -12
 7  715       735              -20
 8  766       785              -19

Average error ~ -7.1 Elo

I know that it sets the upper bound of 99.31% of score, that is, circa 863.3 Elo gain at most. But the average error has improved a lot.

Furthermore, I did not take into account error bars.

Bonus: if I continue giving increasing values of TC to fitted_µ ~ 0.99310185*exp[-0.66489741*exp(-gamma*0.64087232)], I get the next estimated Elo gains:

Code: Select all

Converting fitted_µ into Elo gain (rounding up to the nearest Elo integer):

          Comparison               TC  Fitted Elo
 5120 +  51.2 vs  2560 +  25.6      8  785
10240 + 102.4 vs  5120 +  51.2      9  818 (+33)
20480 + 204.8 vs 10240 + 102.4     10  838 (+20)
40960 + 409.6 vs 20480 + 204.8     11  849 (+11)
81920 + 819.2 vs 40960 + 409.6     12  856 ( +7)

I hope no typos. 818 (+33) should be understood as 818 - 785 = +33 Elo in (10240 + 102.4 vs 5120 + 51.2) and +818 Elo in (10240 + 102.4 vs 10 + 0.1).

It might be interesting to fit win ratio, draw ratio and lose ratio in similar ways.

Last but not least: thank you very much, Andreas.

Regards from Spain.

Ajedrecista.

[corres]

Thanks for correction.
But where are the 1500 opening positions?
I should like to know that how you can calculate the Elo number from a self play match. Elo number is always a relative number. But in the case of self play what is the basic point?
I think from your post some facts are missing still.
Robert