Crafty tests show that Software has advanced more.

[Don]

I didn't really expect that Bob's test would show this as I consider his test rather biased in favor of hardware. Nevertheless, it is still showing that software is a bigger contributor to computer chess advancement over the years than hardware is.

Here are some of his intermediate results:

Code: Select all

   Crafty-23.4        2703    4    4 30000   66%  2579   22% 
   Crafty-23.3        2693    4    4 30000   65%  2579   22% 
   Crafty-23.1        2622    4    4 30000   55%  2579   23% 
   Glaurung 2.2       2606    3    3 60277   46%  2636   22% 
   Toga2              2599    3    3 60275   45%  2636   23% 
   Fruit 2.1          2501    3    3 60248   32%  2636   21% 
   Glaurung 1.1 SMP   2444    3    3 60267   26%  2636   17% 
   Crafty-10.18       2326   19   19  1327   20%  2580   14% 

Here is the calculation to show that software is the bigger contributor:

It's well known that each hardware doubling is worth about 60 ELO of rating improvement. (For example Crafty running on a quad is almost exactly 100 ELO stronger than the single processor equivalent program.)

Bob's test shows that Crafty gained 377 ELO with small error margins. Bob agreed that we should add about 300 ELO to represent true Software advancement because Rybka 4 represents the state of the art in 2010 and it's over 300 ELO stronger than Crafty.

So this test estimates that we have gained 377 + 300 = 677 ELO over a 15 year period.

So the question is how much speed do we need in order to gain 677 ELO if a doubling is worth 60 ELO?

677 / 60 = 11.3 doublings. 11.3 doublings is a factor of 2521. We need a computer well over 2,500 times faster to get 677 ELO.

Bob estimated that hardware increased only 1500 times. Therefore more of the improvement has come from software than hardware using his estimates of hardware improvements.

I would like to mention that I believe Bob's numbers are flawed for several reasons I will briefly outline here and in fact the software is even MORE than Bob estimates.

The first reason is that his numbers do not reconcile with a test I did using Rebel. We compared rebel on old and new hardware. The 1 processor speedup for Rebel is about 100 to 1. Allowing for running on a Octal, you could multiply this by 8 to get 800 to 1. For chess, an octal does NOT give you a true 800 to 1 speedup but Bob is using the Nodes per second calculation anyway. This number still disagrees with Bob's number by about 2 to 1.

Another reason Bob's numbers are distorted is that he decided arbitrarily which machines should be compared. It's a question of defining something to remain a constant such as price, form factor, etc. For example we could say that anything you can purchase for less than 1000 bucks, or anything that is called a "workstation" and that you can easily move around. Of all the possible things to remain constant and with much hand waving he decided the constant should be that it must be Intel hardware. Of all the possible things to compare, this is the one that exaggerates the difference the most. In 1995 more powerful machines were available than the P90, so calling the P90 state of the art is a joke. But calling the i7 state of the art is not.

In order to measure the hardware difference Bob chose to use 2 different versions of Crafty, both of which are optimized to run on 64 bit systems. As coincidence would have it, Crafty runs on 64 bit hardware and looks especially good on 64 bit hardware. So he looks at some log files and eventually produces the number 1500 as the value for how much hardware has advanced over the last 15 years and claims he is being generous to do that. The log files show the speed of a 1995 Crafty running on 32 bit hardware. But even back then Crafty was designed to run on a 64 bit machine.


Moores law is a much perverted and misquoted and reformulated statement of how quickly transistor density changes over the years. I think Moore said that density doubles every 18 months and then way back in 1975 modified his own "law" to every 2 years. It has often been loosely translates that performance doubles every 18 months. This was actually a reformalation based on observation by an Intel colleague of Moore's. In fact, performance on average does NOT double every 18 months, it takes longer. (I have NEVER seen a doubling in performance when I upgrade even every 2 or 3 years although sometimes it's close.)

So Bob's estimate is not in harmony with this (admittedly crude) rule of thumb that nevertheless is widely accepted. Over 15 years even if you assume a full doubling every 18 months you would get 1024 improvement. I think almost everyone things 18 months is on the very generous side.

[Don]

I didn't really expect that Bob's test would show this as I consider his test rather biased in favor of hardware. Nevertheless, it is still showing that software is a bigger contributor to computer chess advancement over the years than hardware is.

Here are some of his intermediate results:

Code: Select all

   Crafty-23.4        2703    4    4 30000   66%  2579   22% 
   Crafty-23.3        2693    4    4 30000   65%  2579   22% 
   Crafty-23.1        2622    4    4 30000   55%  2579   23% 
   Glaurung 2.2       2606    3    3 60277   46%  2636   22% 
   Toga2              2599    3    3 60275   45%  2636   23% 
   Fruit 2.1          2501    3    3 60248   32%  2636   21% 
   Glaurung 1.1 SMP   2444    3    3 60267   26%  2636   17% 
   Crafty-10.18       2326   19   19  1327   20%  2580   14% 

Here is the calculation to show that software is the bigger contributor:

It's well known that each hardware doubling is worth about 60 ELO of rating improvement. (For example Crafty running on a quad is almost exactly 100 ELO stronger than the single processor equivalent program.)

Bob's test shows that Crafty gained 377 ELO with small error margins. Bob agreed that we should add about 300 ELO to represent true Software advancement because Rybka 4 represents the state of the art in 2010 and it's over 300 ELO stronger than Crafty.

So this test estimates that we have gained 377 + 300 = 677 ELO over a 15 year period in SOFTWARE improvements ALONE.

So the question is how much speed do we need in order to gain 677 ELO if a doubling is worth 60 ELO?

677 / 60 = 11.3 doublings. 11.3 doublings is a factor of 2521. We need a computer well over 2,500 times faster to get 677 ELO.

Bob estimated that hardware increased only 1500 times. Therefore more of the improvement has come from software than hardware using his estimates of hardware improvements.

I would like to mention that I believe Bob's numbers are flawed for several reasons I will briefly outline here and in fact the software is even MORE than Bob estimates.

The first reason is that his numbers do not reconcile with a test I did using Rebel. We compared rebel on old and new hardware. The 1 processor speedup for Rebel is about 100 to 1. Allowing for running on a Octal, you could multiply this by 8 to get 800 to 1. For chess, an octal does NOT give you a true 800 to 1 speedup but Bob is using the Nodes per second calculation anyway. This number still disagrees with Bob's number by about 2 to 1.

Another reason Bob's numbers are distorted is that he decided arbitrarily which machines should be compared. It's a question of defining something to remain a constant such as price, form factor, etc. For example we could say that anything you can purchase for less than 1000 bucks, or anything that is called a "workstation" and that you can easily move around. Of all the possible things to remain constant and with much hand waving he decided the constant should be that it must be Intel hardware. Of all the possible things to compare, this is the one that exaggerates the difference the most. In 1995 more powerful machines were available than the P90, so calling the P90 state of the art is a joke. But calling the i7 state of the art is not.

In order to measure the hardware difference Bob chose to use 2 different versions of Crafty, both of which are optimized to run on 64 bit systems. As coincidence would have it, Crafty runs on 64 bit hardware and looks especially good on 64 bit hardware. So he looks at some log files and eventually produces the number 1500 as the value for how much hardware has advanced over the last 15 years and claims he is being generous to do that. The log files show the speed of a 1995 Crafty running on 32 bit hardware. But even back then Crafty was designed to run on a 64 bit machine.


Moores law is a much perverted and misquoted and reformulated statement of how quickly transistor density changes over the years. I think Moore said that density doubles every 18 months and then way back in 1975 modified his own "law" to every 2 years. It has often been loosely translates that performance doubles every 18 months. This was actually a reformalation based on observation by an Intel colleague of Moore's. In fact, performance on average does NOT double every 18 months, it takes longer. (I have NEVER seen a doubling in performance when I upgrade even every 2 or 3 years although sometimes it's close.)

So Bob's estimate is not in harmony with this (admittedly crude) rule of thumb that nevertheless is widely accepted. Over 15 years even if you assume a full doubling every 18 months you would get 1024 improvement. I think most people think 18 months is on the very generous side.

I'm not particularly interested in someones rule of thumb, but in this case I think Bob's estimate flys in the face of common sense, since even the "doubling in performance every 18 month" version of Moore's law is not accurate, especially in recent years where it continues to slow every year.

[Michael Sherwin]

Another valid test to give data on this issue is to play Komodo against a time handicapped R4 to see what handicap would make them even.

[mhull]

I wonder why you don't bring up the fact that Rebel could have been compiled to run on faster hardware in 1995, but instead you continue with your P90 comparison with modern hardware. It seems like you are applying a double-standard to crafty in this instance.

Even so, the point you are making about could-haves in 1995 has already been addressed here. And your Rebel test is a tacit admission of the validity of that point, IMO. The point is that it matters not what could have been. It's totally beside the point.

For instance, I still have a Crafty 9.x executable that I used to run on my Mac Plus (68000 at 8 Mhz). I still have this and it still runs on my old 68030 Mac. A comparison between crafty on that hardware could be made (in principle) with modern compile on the latest Intel machine (if we had the source for that version). Performance could be compared to a modern compile, just like with the 1995 Intel-targeted crafty that Bob is doing now. ELO could be estimated for both platforms and the HW speed delta determined.

So I confess my own failure to understand what more this particular hobby horse you are riding can possibly tell us, except that you are very upset about some something beyond my feeble grasp.

[Don]

Another valid test to give data on this issue is to play Komodo against a time handicapped R4 to see what handicap would make them even.

Rybka is much stronger than Komodo. The two issues are software and hardware so this test would have to somehow relate the two, right?

I think people overestimate how much hardware has to improve to get a small improvement in ELO strength. If you can find 2 ELO per month, you are just about staying even with hardware advancement. You can look back 10 years and notice that computers are enormously faster than they used to be and attribute all of that to why they play stronger, but the 2 ELO per month is apparently just as much of the reason as Bob's experiments indicate.

[bob]

I didn't really expect that Bob's test would show this as I consider his test rather biased in favor of hardware. Nevertheless, it is still showing that software is a bigger contributor to computer chess advancement over the years than hardware is.

Here are some of his intermediate results:

Code: Select all

   Crafty-23.4        2703    4    4 30000   66%  2579   22% 
   Crafty-23.3        2693    4    4 30000   65%  2579   22% 
   Crafty-23.1        2622    4    4 30000   55%  2579   23% 
   Glaurung 2.2       2606    3    3 60277   46%  2636   22% 
   Toga2              2599    3    3 60275   45%  2636   23% 
   Fruit 2.1          2501    3    3 60248   32%  2636   21% 
   Glaurung 1.1 SMP   2444    3    3 60267   26%  2636   17% 
   Crafty-10.18       2326   19   19  1327   20%  2580   14% 

Here is the calculation to show that software is the bigger contributor:

It's well known that each hardware doubling is worth about 60 ELO of rating improvement. (For example Crafty running on a quad is almost exactly 100 ELO stronger than the single processor equivalent program.)

Bob's test shows that Crafty gained 377 ELO with small error margins. Bob agreed that we should add about 300 ELO to represent true Software advancement because Rybka 4 represents the state of the art in 2010 and it's over 300 ELO stronger than Crafty.

So this test estimates that we have gained 377 + 300 = 677 ELO over a 15 year period.

So the question is how much speed do we need in order to gain 677 ELO if a doubling is worth 60 ELO?

677 / 60 = 11.3 doublings. 11.3 doublings is a factor of 2521. We need a computer well over 2,500 times faster to get 677 ELO.

Bob estimated that hardware increased only 1500 times. Therefore more of the improvement has come from software than hardware using his estimates of hardware improvements.

I would like to mention that I believe Bob's numbers are flawed for several reasons I will briefly outline here and in fact the software is even MORE than Bob estimates.

The first reason is that his numbers do not reconcile with a test I did using Rebel. We compared rebel on old and new hardware. The 1 processor speedup for Rebel is about 100 to 1. Allowing for running on a Octal, you could multiply this by 8 to get 800 to 1. For chess, an octal does NOT give you a true 800 to 1 speedup but Bob is using the Nodes per second calculation anyway. This number still disagrees with Bob's number by about 2 to 1.

Another reason Bob's numbers are distorted is that he decided arbitrarily which machines should be compared. It's a question of defining something to remain a constant such as price, form factor, etc. For example we could say that anything you can purchase for less than 1000 bucks, or anything that is called a "workstation" and that you can easily move around. Of all the possible things to remain constant and with much hand waving he decided the constant should be that it must be Intel hardware. Of all the possible things to compare, this is the one that exaggerates the difference the most. In 1995 more powerful machines were available than the P90, so calling the P90 state of the art is a joke. But calling the i7 state of the art is not.

Fine. For state of the art I choose deep blue. 1,000,000,000 nodes per second. Pick _any_ 1995 platform you want and let's compare speedup. Or turn it around and I pick a Cray T932 which is more computer than any single chip PC today in any type of measurement. So we have 0 hardware improvement.

Or we use the machine that _everybody_ was using in 1995, which was Intel/windows, and we use the machine that _everybody_ is using today, which is the i3/i5/i7.

Personally, I have no problem determining which is the test to run. Everybody that has run on a T932, raise your hand. Looking around I see _one_ hand up. Everyone that has run on the big SP cluster with special-purpose chess processors, raise your hand. Again, I see _one_ hand up.

The point is that if you ask _anyone_ here what they were using in 1995, from the SSDF list, to ICC, to WMCCC/WCCC events, the most common answer, by probably 30-1 is going to be an Intel PC. That's the machine class almost everyone today is using as well. So the noise about the alphas and such is pure nonsense. Because if we include alphas we have to include every other rarely used box, of which there are many, and they were/are extremely fast. And extremely expensive...

In order to measure the hardware difference Bob chose to use 2 different versions of Crafty, both of which are optimized to run on 64 bit systems.

Jeez, Don, can't you at least read and get this right? My speed comparison was with crafty 10.x from 1995. I had numbers for the P5/133, I ran it on my hardware to get the speedup today. Not two different versions. The _exact_ same version. With about two dozen changes to add the xboard protocol changes to make it work on my cluster. Not a single change to the engine itself.

Why do you insist on continuing to make such a stupid statement (two different versions.) It is _clearly_ false. I doubt a single person here (perhaps excepting yourself) has somehow misunderstood that specific detail, which has been explained enough for anyone to finally see the light.

To emphasize: version 10.x was run in 1995 on 1995 hardware. I had a few test positions that backed up my 30K recollection. I ran the same few positions using that same version, but used my E5345 (single cpu) machine. It ran at 4M nodes per second. I ran crafty 23.4 on the same positions, same processor. Almost exactly the same NPS. I posted the numbers yesterday.

Again: 10.x on P5/133 searches 30K. On a P5/90, 20K. On an E5345 single CPU, 4M. It will take a little work to get the smp search to work, because the pthread library changed from way back and it doesn't compile cleanly. In addition, the lock stuff (xchg lock) has to be modified to work with 64 bit stuff as it uses the wrong register names. Those versions scaled perfectly with NPS, as the current version does. So 30M+ is the expected number. I will verify this once I get the pthread stuff to working over the next couple of days.

Should I repeat it one more time? Not two different versions. _same_ version.

As coincidence would have it, Crafty runs on 64 bit hardware and looks especially good on 64 bit hardware. So he looks at some log files and eventually produces the number 1500 as the value for how much hardware has advanced over the last 15 years and claims he is being generous to do that. The log files show the speed of a 1995 Crafty running on 32 bit hardware. But even back then Crafty was designed to run on a 64 bit machine.

That is a false statement. Crafty was designed to use 64 bit values for the bitboards. It was _designed_ to run on 32 bit hardware, which was what we had in the PC world back then. You only have to look at the move generation stuff (COMPACT_ATTACKS, USE_SPLIT_SHIFTS, etc) that was explicitly designed to work efficiently on 32 bit boxes. Yes it gains some on 64 bit hardware. But in 1995 it was most certainly designed to run well on 32 bit hardware.

Hell, I'd bet you dollars to donuts you had 64 bit stuff in your 1995 code. Hashing, anyone? I've always used 64 bit hashing. Back then I did it as two 32 bit chunks, but it would clearly fit 64 bit hardware better. So was _your_ stuff designed for 64 bits only? Didn't think so.

Moores law is a much perverted and misquoted and reformulated statement of how quickly transistor density changes over the years. I think Moore said that density doubles every 18 months and then way back in 1975 modified his own "law" to every 2 years. It has often been loosely translates that performance doubles every 18 months. This was actually a reformalation based on observation by an Intel colleague of Moore's. In fact, performance on average does NOT double every 18 months, it takes longer. (I have NEVER seen a doubling in performance when I upgrade even every 2 or 3 years although sometimes it's close.)

So Bob's estimate is not in harmony with this (admittedly crude) rule of thumb that nevertheless is widely accepted. Over 15 years even if you assume a full doubling every 18 months you would get 1024 improvement. I think almost everyone things 18 months is on the very generous side.

Keep saying that to yourself enough, and perhaps you will believe it. But I have no "estimate". I have an absolute measured value. Taking the P5/90 on one end, and a 6-core i7 on the other, the speed increase for Crafty is 1500x. I did not claim that was the speed gain for any other program. I don't care about any other program. I did not 'estimate' a thing, I simply took out my "ruler" and measured both as accurately as possible. What you are talking about is something that might have been typed by a roomfull of monkeys, because it is valid words, and somewhat valid grammatical constructions, but the meaning is missing.

So get off the "estimation" and "fabrication" and "exaggeration" bandwagon and offer something useful and logical. I've explained my numbers. Feel free to shoot either the 22K or the 30M numbers down. We can certainly get a 3rd party to verify the 10.x on current hardware. We've already had confirmation by someone running crafty and seeing 22K nps on I think a P5/100mhz. Which is right in line with 20K at 90mhz and 30K at 133mhz.

So shoot at what you think is wrong, but don't try to restate what I have done, I have been precise in what I have measured. And it is nothing at all related to what you are claiming I have measured.

I am also not yet ready to grant Rybka another +300 Elo on software improvements. It may well be that Crafty has a serious flaw somewhere. Ideal would be to run rybka thru the same test I am doing. But that's not an option for me since there is no source available.

[Dirt]

I think people overestimate how much hardware has to improve to get a small improvement in ELO strength. If you can find 2 ELO per month, you are just about staying even with hardware advancement.

Good point. I realized that software gains were probably more important than hardware speed-ups a couple of years ago when Uri Blass pointed out that the SSDF results implied it. Kudos to the programmers.

[Don]

I wonder why you don't bring up the fact that Rebel could have been compiled to run on faster hardware in 1995, but instead you continue with your P90 comparison with modern hardware. It seems like you are applying a double-standard to crafty in this instance.

I just don't understand your point. Rebel was probably compiled in 1995 to be as fast as it could be at the time, so how could Ed have compiled it to be even faster? I'm sure he would have if he could have. Are you claiming that he didn't know what he was doing?

I explained the 64 bit vs 32 bit issue and someone else explained it better than I did.

By the way, the 100 to 1 figure is not accurate and I already admitted that in one of my posts. To be a (more or less) fair comparison, Rebel should be compiled on an i7 with a modern compiler. Then it would show more than 100 to 1 but it would surely still be less than 200 to 1. Unfortunately we don't have the means to do this so I went with what I could get. However Bob has the means with his old Crafty version, unfortunately that overstates the hardware difference by breaking Bob's own rule that to be fair the program you are comparing should be optimized equally well for the hardware they are running on.

I think you might understand what I am saying if I cast the issue differently. Suppose you purposely design a program to run especially slow on a 32 bit computer but to run fast on a 64 bit computer? Wouldn't it look especially good going to 64 bits?

So basically ANY 64 bit program is designed to run especially slow on a 32 bit operating system and does not take full advantage of a 32 bit computer. The authors CHOSE that design knowing full well that it was not the best way to write a program that runs on a 32 bit computer. Bob Hyatt knew back in 1995 that Crafty would not run as well on a 32 bit computer but was smart enough to design for the future - knowing that in the short term his program would take a hit.

That's a very reasonable decision, but it makes comparisons on old hardware off by something like 3 to 2.

Even so, the point you are making about could-haves in 1995 has already been addressed here. And your Rebel test is a tacit admission of the validity of that point, IMO. The point is that it matters not what could have been. It's totally beside the point.

I don't think you really get it.

For instance, I still have a Crafty 9.x executable that I used to run on my Mac Plus (68000 at 8 Mhz). I still have this and it still runs on my old 68030 Mac. A comparison between crafty on that hardware could be made (in principle) with modern compile on the latest Intel machine (if we had the source for that version). Performance could be compared to a modern compile, just like with the 1995 Intel-targeted crafty that Bob is doing now. ELO could be estimated for both platforms and the HW speed delta determined.

Bob did something like that already. I think a reasonably valid way for Bob to recalibrate is to make a 32 bit version of the NEW program and compare the NPS to the old logs. A 32 bit program is not optimal on 64 bit hardware, but a program written to be optimial only on a 64 bit platform is not optimal on 32 bit hardware either, so this test would be closer to the truth.

But does it really matter now? Bob has been proved wrong using his own data so it's a moot point.

So I confess my own failure to understand what more this particular hobby horse you are riding can possibly tell us, except that you are very upset about some something beyond my feeble grasp.

I'm not upset at all - I'm only slightly frustrated that I cannot seem to explain a simple concept.

[bob]

I think people overestimate how much hardware has to improve to get a small improvement in ELO strength. If you can find 2 ELO per month, you are just about staying even with hardware advancement.

Good point. I realized that software gains were probably more important than hardware speed-ups a couple of years ago when Uri Blass pointed out that the SSDF results implied it. Kudos to the programmers.

Now, if only that math were correct, everyone would be happy. But 2 Elo per month is nonsense.

Typically we have seen 60-70 Elo for every doubling of speed for many years. Thompson and then Berliner ran these kinds of tests several times. If you figure 2x speed every 2 years, which is actually slow (for some of us, anyway, more in a minute) that is about 30-35 Elo per year purely on speed. Or 3 Elo per month.

Or, if you take the speedup numbers I have actually measured using 10.18, you get 1500x. But let me back off of that a bit. 250x is pure single-cpu speed gain, going from 20K in 1995 to 4M on one cpu in 2006 hardware, more on an i7 but I don't have any single CPU numbers so let's stop at 4M. 20K to 4M is 200x faster. using 8 cpus, Crafty's speedup is roughly 1 + 7 * .7 = 5.9 or a bit better. Call that 6x. So 200x times 6 = 1200x effective speedup, which doesn't give credit for all cores, and factors in parallel search efficiency. 1200x is 10+ doublings. That will definitely be more than 360 Elo for me. In a few days I can say exactly how much more once I figure out a way to test at 20K on modern hardware, and do it correctly...

[bob]

I think people overestimate how much hardware has to improve to get a small improvement in ELO strength. If you can find 2 ELO per month, you are just about staying even with hardware advancement.

Good point. I realized that software gains were probably more important than hardware speed-ups a couple of years ago when Uri Blass pointed out that the SSDF results implied it. Kudos to the programmers.

Now, if only that math were correct, everyone would be happy. But 2 Elo per month is nonsense.

Typically we have seen 60-70 Elo for every doubling of speed for many years. Thompson and then Berliner ran these kinds of tests several times. If you figure 2x speed every 2 years, which is actually slow (for some of us, anyway, more in a minute) that is about 30-35 Elo per year purely on speed. Or 3 Elo per month.

Or, if you take the speedup numbers I have actually measured using 10.18, you get 1500x. But let me back off of that a bit. 250x is pure single-cpu speed gain, going from 20K in 1995 to 4M on one cpu in 2006 hardware, more on an i7 but I don't have any single CPU numbers so let's stop at 4M. 20K to 4M is 200x faster. using 8 cpus, Crafty's speedup is roughly 1 + 7 * .7 = 5.9 or a bit better. Call that 6x. So 200x times 6 = 1200x effective speedup, which doesn't give credit for all cores, and factors in parallel search efficiency. 1200x is 10+ doublings. That will definitely be more than 360 Elo for me. In a few days I can say exactly how much more once I figure out a way to test at 20K on modern hardware, and do it correctly...

BTW, taking that 10+ doublings in 15 years suggests that _some_ of us have been seeing Moore's law tick right along as predicted. Not everybody, of course, but some.