I recently converted my java chess engine CuckooChess to C++ (and renamed it to Texel). As a result the engine became approximately twice as fast. Initial tests suggest that the elo gain is much larger than what would have been expected according to the old rule that you gain 50-70 elo points for each doubling of the speed.
In an attempt to understand what is going on I played a time odds match using three different engines.
Match conditions
* Three engines tested: stockfish, crafty, texel
* Each engine plays at three different time controls:
1: 60 moves in 6 seconds
2: 60 moves in 12 seconds
3: 60 moves in 24 seconds
* Ponder off
* One CPU
* 128MB transposition table
* Openings selected randomly from a set of 7316 book lines, each line 10 ply deep.
* Each opening line played twice with colors reversed second time.
* Three test computers used, all running 64 bit linux.
- Core2 Quad 2.4GHz
- Core i7 870, 2.93GHz
- Core2 Duo, 2.6GHz
* Tournament where each engine plays all other engines.
* Total number of games: 14976
* Total number of games per engine: 3328
* Total number of games between each pair of engines: 416
* Total number of time losses: 1
Results
Rating computations were performed using bayeselo. All 14976 games together gives the following rating table:
Code: Select all
Rank Name Elo + - games score oppo. draws
1 stockfish3 372 14 14 3328 91% -47 14%
2 stockfish2 228 12 11 3328 78% -28 21%
3 stockfish1 46 11 10 3328 56% -6 23%
4 texel3 32 11 11 3328 55% -4 21%
5 crafty3 30 11 10 3328 55% -4 23%
6 crafty2 -76 11 10 3328 41% 9 23%
7 texel2 -116 10 11 3328 36% 14 21%
8 crafty1 -222 11 12 3328 23% 28 19%
9 texel1 -294 12 13 3328 16% 37 14%
Code: Select all
1->2 2->3
stockfish 182 144
crafty 146 106
texel 178 148
Code: Select all
s3 s2 s1 c3 c2 c1 t3 t2 t1
stockfish3 0 134 344 330 448 576 334 500 624
stockfish2 -134 0 178 224 336 396 168 342 470
stockfish1 -344 -178 0 16 108 268 50 160 308
crafty3 -330 -224 -16 0 114 270 -2 136 324
crafty2 -448 -336 -108 -114 0 174 -98 42 190
crafty1 -576 -396 -268 -270 -174 0 -228 -104 76
texel3 -334 -168 -50 2 98 228 0 162 372
texel2 -500 -342 -160 -136 -42 104 -162 0 196
texel1 -624 -470 -308 -324 -190 -76 -372 -196 0
Code: Select all
s3 s2 s1 c3 c2 c1 t3 t2 t1
stockfish 2->3 134 134 166 106 112 180 166 158 154
stockfish 1->2 210 178 178 208 228 128 118 182 162
crafty 2->3 118 112 92 114 114 96 96 94 134
crafty 1->2 128 60 160 156 174 174 130 146 114
texel 2->3 166 174 110 138 140 124 162 162 176
texel 1->2 124 128 148 188 148 180 210 196 196
I believe the "50-70 elo increase per speed doubling" estimate originally comes from "How Computers Play Chess" by David Levy and Monty Newborn in 1991. I don't have that book, so I don't know what data they based the estimate on. I guess they used longer time controls than I did, but given 20 years of hardware improvement, it is not unlikely that my tests generated larger search trees anyway.
In my last table, most values are significantly larger than 50-70. In fact only one value (crafty 1->2 vs stockfish2) falls within the 50-70 range. It is worth noting that each value in that table is calculated from only 416 games, so the error margin according to bayeselo is about 30. However, if the true values were within 50-70, getting the result I got would be extremely unlikely.
Are there any newer estimates of elo versus speed? I would not be surprised if todays engines with recursive null moves, late move reductions and other pruning techniques behave quite differently from the ones from 20 years ago.