Software gain. I am comparing Crafty 23.4, the highest-rated version of Crafty so far, to Crafty 10.18 which was a 1995 program. First, the data.
Code: Select all
Crafty-23.4-2 2851 3 3 30000 66% 2728 22% Crafty-23.4-1 2848 3 3 30000 66% 2728 22% Crafty-10.18-1 2491 4 4 30000 22% 2728 14% Crafty-10.18-2 2490 4 4 30000 22% 2728 14%
Hardware gain. This part is easy. On a P5/90, we searched 20K nodes per second, today on the best hardware platform, a 6-core i7, the same version of Crafty (10.18) runs at 30M nodes per second using SMP search. With PGO it gains another 10%. But for
First, compute the actual speedup, since the SMP search speed scales perfectly with respect to NPS, but not that well when you measure time to solution. For SMP efficiency, after measuring thousands of positions over the years, a good estimate for parallel speedup is:
speedup = 1 + (NCPUS - 1) * 0.7
For the 6 core box, we get 4.5x increase (rather than 6x) for the above. If you take the 30M number and divide by 6 cpus, you get 5M nodes per second per CPU. Multiplying by the 4.5x SMP speedup, we get 22.5M nodes per second. Divide this by the 1995 speed of 20K and we are a little over the 1000x faster number I have quoted repeatedly. For simplicity, call this 1024x.
Now, what does this do? Historically we have had several estimates for the Elo gain for each doubling of CPU speed. Way back, it was 100. More recently I have been seeing 50-70, so I thought I would try to measure this as easily as possible. I decided to take 10.18, and run it thru the gauntlet again, but this time gave it 1/2 the normal time, leaving the opponents using the normal time. I then ran this two more times, but used 1/4 of the normal time.
Here's the data:
Code: Select all
Crafty-10.18-1 2491 4 4 30000 22% 2728 14% Crafty-10.18-2 2490 4 4 30000 22% 2728 14% Crafty-10.18-3 2415 4 4 30000 16% 2728 11% Crafty-10.18-4 2413 4 4 30000 15% 2728 11% Crafty-10.18-5 2325 5 5 30000 10% 2728 8% Crafty-10.18 2325 5 5 28840 10% 2728 8%
If you look carefully, -3 and -4 are 76 elo lower (average) than -1/-2. Then -5 and -6 are 88 lower. I suspect this trend will continue, in that if there is a diminishing return for going faster and faster, then there must be a steadily increasing loss for going slower and slower. If we pick a pretty rough average of 80 for each doubling for 10.18, and if you compute log2(1024) = 10, or 10 doublings in hardware speed running Crafty, since 1995. At 80 Elo per doubling, we get +800. I don't particularly trust the +360 or this +800, but I am reasonably certain that the ratio would not change by much.
This is roughly 2-1, 2 parts hardware to 1 part software. I wasn't quite sure where this would end up, but this is certainly within the high/low bounds I would have guessed.
While it is possible to take the hardware speeds all the way back to 1995, it is not easy. I need to find some very weak opponents because when I was trying to test the new skill settings, I could not get below 1600 or so, because the opposition I use is too strong. That is not an easy thing to do, and to be honest, it is a lot of effort for absolutely zero return. I've decided that this rough (but pretty accurate) assessment is close enough for the time being. Anyone else can test whatever program they want. More data would be interesting. But it is a lot of work for no return.
The rather ridiculous title "Crafty tests show that Software has advanced more." is totally misleading, and completely wrong. My results suggest a 1/3-2/3 split, 1/3 for software, 2/3 for hardware. I know that the arguments will continue. I've provided the best data I can. Hopefully we will see more real data and less speculation and hot air as this continues...