ccrl hardware adjustment question

[lkaufman]

The ccrl 40/15 ratings use time controls adapted to a specific I7 model. My question is: How many times longer would you set the time control for a test if it ran on the same hardware as was used in the Fritz vs. Kramnik and Kasparov matches in 2002, 2003, and 2006, assuming both used 4 cores/4 threads? In other words, how much faster per core is the hardware of your reference i7 than the hardware they used in each of those years? I don't expect an exact answer, just a rough estimate for each year. That would give us a way of estimating how much to add to your numbers to get estimates of how today's engines would perform on the reference hardware against the top players today.

[kasinp]

I remember analyzing a similar question, perhaps it may help calibrate the CCRL comparison:

In 2003 Junior played Kasparov on a quad Intel Xeon 1.9GHz. My analysis (coupled with anecdotal information about Junior's nps) gave it approximately 50-55% of the same Junior version on a stock quad Q6600 2.4GHz.

In 2007 Fritz played Kramnik on what I believe was a dual Xeon DC 5160 (4 CPUs in total), and was approximately 25% faster than Q6600 2.4GHz.

Regards,
Peter

[lkaufman]

I remember analyzing a similar question, perhaps it may help calibrate the CCRL comparison:

In 2003 Junior played Kasparov on a quad Intel Xeon 1.9GHz. My analysis (coupled with anecdotal information about Junior's nps) gave it approximately 50-55% of the same Junior version on a stock quad Q6600 2.4GHz.

In 2007 Fritz played Kramnik on what I believe was a dual Xeon DC 5160 (4 CPUs in total), and was approximately 25% faster than Q6600 2.4GHz.

Regards,
Peter

Thanks. The reference hardware for CCRL is Intel i7-4770K. How much faster is that for chess compared to the Q6600 you mention on same number of cores/threads?

[Modern Times]

Thanks. The reference hardware for CCRL is Intel i7-4770K. How much faster is that for chess compared to the Q6600 you mention on same number of cores/threads?

I'd guess twice as fast but not really sure.

Pretty sure Graham has both the reference i7-4770k and a Q6600 so he could probably run Komodo bench on both and tell you. I still have a QX6700 from 2009 which I haven't used for chess for a number of years, which is the same Kentsfield architecture but a slightly faster clock (2.66 vs 2.4). If you want a Komodo bench on that machine I could run it.

[lkaufman]

Thanks. The reference hardware for CCRL is Intel i7-4770K. How much faster is that for chess compared to the Q6600 you mention on same number of cores/threads?

I'd guess twice as fast but not really sure.

Pretty sure Graham has both the reference i7-4770k and a Q6600 so he could probably run Komodo bench on both and tell you. I still have a QX6700 from 2009 which I haven't used for chess for a number of years, which is the same Kentsfield architecture but a slightly faster clock (2.66 vs 2.4). If you want a Komodo bench on that machine I could run it.

Sure, if you can give comparative bench for your qx6700 and the i7-4770k, we can adjust by the 2.66 to 2.4 ratio, and that should tell us the ratio to use with sufficient accuracy.

[kasinp]

I remember analyzing a similar question, perhaps it may help calibrate the CCRL comparison:

In 2003 Junior played Kasparov on a quad Intel Xeon 1.9GHz. My analysis (coupled with anecdotal information about Junior's nps) gave it approximately 50-55% of the same Junior version on a stock quad Q6600 2.4GHz.

In 2007 Fritz played Kramnik on what I believe was a dual Xeon DC 5160 (4 CPUs in total), and was approximately 25% faster than Q6600 2.4GHz.

Regards,
Peter

Thanks. The reference hardware for CCRL is Intel i7-4770K. How much faster is that for chess compared to the Q6600 you mention on same number of cores/threads?

i7 4790 Fritz mark (4 threads) = 12,155
Q6600 Fritz mark (4 threads) = 6,512

The difference between i7-4790 and i7-4770K is negligible (Passmark 7207 vs. 7076).
To me it looks like i7-4770K is approximately 1.83 times faster than Q6600.

Peter

[MikeB]

The ccrl 40/15 ratings use time controls adapted to a specific I7 model. My question is: How many times longer would you set the time control for a test if it ran on the same hardware as was used in the Fritz vs. Kramnik and Kasparov matches in 2002, 2003, and 2006, assuming both used 4 cores/4 threads? In other words, how much faster per core is the hardware of your reference i7 than the hardware they used in each of those years? I don't expect an exact answer, just a rough estimate for each year. That would give us a way of estimating how much to add to your numbers to get estimates of how today's engines would perform on the reference hardware against the top players today.

Larry for chess processing power, the integer benchmarks are most likely the best proxy for chess strength. With the Q6600 having a stock speed of 2.4 Ghz and with the I7 4790 having a stock speed of 3.6 Ghz , the I7-4790 geekbench benchmark v4.0 integer score using one thead score is approximately ~2.1x of the Q6600 using the same benchmark with both running at their respective stock speeds ( not OC).

[Modern Times]

OK so I was about right with my estimate of twice the speed.

[Graham Banks]

I used 40 moves in 33 minutes repeating on the Q6620 (Crafty benchmark), whereas I use 40 moves in 15 minutes repeating on the i7-4770 (Stockfish benchmark).

[lkaufman]

I used 40 moves in 33 minutes repeating on the Q6620 (Crafty benchmark), whereas I use 40 moves in 15 minutes repeating on the i7-4770 (Stockfish benchmark).

OK, so putting this together with the estimates for the old machines, here's what we have. There were three matches at standard time limit between the top engines (Deep Fritz and Deep Junior) and the two top human players (Kasparov and Kramnik) in the late 2002 thru 2003 period, all of which resulted in even scores. Kasparov (who played 10 games) was about 2830 then, Kramnik (who played 8 games) about 2810. Based on the info in this thread, the CCRL reference hardware is just about four times as fast as the hardware used during these matches. Although it's hard to match up commercial versions and the ones used in these matches exactly, I think that the Fritz matches involved Fritz 7 to Fritz 8, and the Junior one around Junior 9 (not so sure about this one). I think they all ran on four cores. On the CCRL 40/15 list, Fritz 8 is 2701, Junior 9 is 2676, Fritz 7 not listed but presumably around 2660 or so based on the rate of progress at the time. So the average CCRL rating is about 2680, on one thread. In general, running on four threads gives nearly as much elo as a tripling of speed, so the hardware they ran on was about 3/4 of the speed of the reference hardware for one thread on the CCRL list. A 4 to 3 speedup is worth something like 25 elo or so on this list, so the average CCRL rating for the actual hardware used with four cores should be about 25 elo less than the nominal single core values. So the CCRL list "predicts" a performance of about 2655 for the engines. But they actually performed around 2820. So my conclusion is that for engines in the range of the top human players, you should add 165 elo to estimate the ratings these engines would get on the reference hardware at standard time limits vs. top human players. As a cross-check, let's look at the 2006 Fritz vs. Kramnik match, which was won by Fritz 4 to 2. In this case we have a 4 cpu rating for the same version (Fritz 10) of 2830. The hardware speedup is only 2.1/1.25 = 1.68 to 1, maybe 45 elo or so on this list. So the "predicted" rating for the hardware used is 2785. Kramnik's rating was a bit low at the time of the match, but a representative rating for him around this period was probably 2800, so Fritz performed about 2920, 135 elo above the prediction. Note also that Kramnik got more advantages than normal for this match, notably that he could look at the engine's opening book during the game! Anyway not too far off from the 165 figure. So let's say add 160 elo in round numbers to predict standard rating. Note that I'm not getting into the issue of whether rating differences are the same for engines and humans; I know that engine differences are smaller, but CCRL differences are already reduced by using BayesElo, so it's not a huge effect. But I'm not trying to claim that a 3500 CCRL rating means 3660 FIDE, I'm saying that a 2700 engine on the CCRL 40/15 list should be an even match for Carlsen at standard time controls. Fritz 8 happens to be right there, which makes sense since it split with Kasparov in 2003 on four cores, so presumably on an i7 with one core it should be a bit stronger and so a good match for Carlsen now. Other engine around 2700 include Gaviota 0.85, Naum 2, Fruit 2.1, and Crafty 23.2. If Carlsen actually played a standard tournament against these engines, would he come out ahead or behind (I guess we should assume a state of the art opening book)? I'm calling it a tossup.
Now how about rapid chess, since these CCRL ratings are actually rapid ratings? Fortunately, the most official FIDE rapid time limit, 15' + 10" (the one used in the FIDE Rapid championship and also the recent online Magnus Carlsen tournament, won by himself) is quite close in general to the CCRL 40/15' time limit, so I'll treat them as interchangeable. It is very well known that engines perform better vs. humans at Rapid time limits such as these than at standard one, although I don't know of any really solid data on how much better. I'm reasonably confident that the gain is more than one hundred elo, but perhaps not much more, I'll guesstimate 120. If anyone has any data that would make for a better estimate, please share. So if I'm correct, the best estimate of what FIDE rapid rating an engine would get would be its CCRL 40/15 rating plus 280, again assuming we're in or near the human range. Carlsen is 2881 FIDE rapid, so this means that an engine rated 2600 CCRL 40/15 should be a fair match for MC at 15' + 10".
Engines around this level include Gaviota 0.83, Arasan 13.0, Scorpio 1.7, and Ethereal 8.05. These should be tossups at Rapid vs the champ.
Is there any evidence that would suggest that I am either overestimating or underestimating the level of these engines vs. top humans? I could of course include many more old events, but I doubt that would change the conclusions by much. Perhaps I have missed something in either direction.