Made In Heaven class Time Control Comparison

[Aser Huerga]

Hi.
What book/testset has been used?
How much the quality of neutral starting positions is important in your opinion?

150 Early Starting Positions Suite, slightly tunned to avoid transpositions (checked with engine vs same-engine matches), and created as a proportional representation of the most played openings/variations on recent years in high quality human chess tournaments (source TWIC, only 2400+ ELO players) AH_150_Opening_Suite
All positions are played with Switched Colors for a total of 300 games per match

There's a lot of approaches choosing starting positions and I'm not claiming my Opening Suite is better than others, but is a good representation of what a wide range of good players are playing in the recent years.

Cheers.

[PCM72]

Thank you.
I think it's a good opening suite but I'm working to test such suites to merge them in a better tuned suite, using the criteria you mentioned and many other ones too.
BTW, just to give a slightly better "idea of extrapolation" from your graph, I've tuned the last two values to see "proportionally" what happen tripling the time control (there was a doubling followed by a tripling without a correct proportion).

[ouachita]

BTW, just to give a slightly better "idea of extrapolation" from your graph, I've tuned the last two values to see "proportionally" what happen tripling the time control (there was a doubling followed by a tripling without a correct proportion).

Is this a revised chart showing your extrapolation or have you posted same? (It looks the same as the original)

[PCM72]

It looks the same as the original because it's just slightly revised:
the only difference is that the last part of the curves (the last 2 values) are now more "compressed" within the graph.
So, basically it's the same as the original, with a slightly corrected "optical illusion". Of course, with other really demanding works like the original (e.g. repeating the test with other books/op.suites and other time controls), we could see really revised and more informative charts.

[ouachita]

lightly better "idea of extrapolation"

wouldn't the inclusion of TCEC 120+ data be even better

[Modern Times]

wouldn't the inclusion of TCEC 120+ data be even better

Very few games.

[PCM72]

Maybe, though the low # of games and the different versions of Stockfish and Houdini.
Maybe in a few months will be even better including data from CEGT 40120, CCRL 40/40, and LightSpeed too, but merging data from different conditions and "flimsy different" books is a different issue and, maybe, a harder work.

[ouachita]

Very few games.

I understand, but it's the best set of 40/120+ we have available, and this is afterall a chart showing general trend lines. A "Type A" statistician will likely not be comfortable with any of these chart figures.

[Vinvin]

Looking for some volunteers to redo this experience with Komodo 8, Stockfish 201409xx and Houdini 4 !
But with more games for faster TC : 1200 games in 3+1 and 9+3 ; 900 games in 27+9; 600 games for longer TC.

As a Shaun Brewer suggestion, I decided to run my games at different Time Controls to see how the top engines strength change as time increases. Here are the results:

Five i7-3930K CPUs 4.25 GHz
1 core for all engines
Ponder off
1024 Hash
3-4-5 EGTBs (when available) in SSDs

Code: Select all

3'+1" Time Control

   # PLAYER          : RATING  ERROR   POINTS  PLAYED    (%)
   1 Houdini 4       :   32.1   13.3    340.0     600   56.7%
   2 Komodo TCEC     :  -14.4   13.4    282.0     600   47.0%
   3 Stockfish DD    :  -17.6   13.8    278.0     600   46.3%


9'+3" Time Control

   # PLAYER          : RATING  ERROR   POINTS  PLAYED    (%)
   1 Stockfish DD    :   16.6   14.1    320.5     600   53.4%
   2 Houdini 4       :    8.1   13.5    310.0     600   51.7%
   3 Komodo TCEC     :  -24.7   13.4    269.5     600   44.9%

27'+9" Time Control

   # PLAYER          : RATING  ERROR   POINTS  PLAYED    (%)
   1 Stockfish DD    :   22.6   14.1    328.0     600   54.7%
   2 Houdini 4       :    5.7   13.4    307.0     600   51.2%
   3 Komodo TCEC     :  -28.3   13.4    265.0     600   44.2%

54'+18" Time Control

   # PLAYER          : RATING  ERROR   POINTS  PLAYED    (%)
   1 Stockfish DD    :   11.4   14.2    314.0     600   52.3%
   2 Houdini 4       :    0.4   13.7    300.5     600   50.1%
   3 Komodo TCEC     :  -11.8   13.6    285.5     600   47.6%

90'+30" Time Control

   # PLAYER          : RATING  ERROR   POINTS  PLAYED    (%)
   1 Stockfish DD    :   10.5   12.9    313.0     600   52.2%
   2 Komodo TCEC     :    0.8   13.2    301.0     600   50.2%
   3 Houdini 4       :  -11.3   13.1    286.0     600   47.7%

I want to thanks Adam Hair for his help in the presentation of graph and results.

( All the games can be downloaded here: TTC_All_Games )