Hardware vs Software

[krazyken]

Well, the thing is that we aren't talking about hardware vs software here. The way this discussion has been going we have gone three categories:
1. Hardware
2. Software techniques
3. all other software improvements.

When it is limited to the first two as Bob is doing, it is clear that hardware is the dominating factor. What isn't so clear is whether or not 2 & 3 combined keeps up with 1. It's not uncommon to see a top level program get updated on the same hardware with a significant gain in strength.

I don't agree with the category definitions. I would use:

(1) hardware
(2) software
(3) software features only possible because of hardware improvements

(1) is clear in what it means.

(2) is also clear in what it means.

(3) is the interesting case. For example, null move R=3 would fail miserably on a pentium-60mhz program because the depth would be so shallow, null-move errors would be rampant. But this category is _still_ a direct result of hardware improvements. The first version of "blitz" searched around 1 node per second in 1970. By 1980 on a Cray, we were doing 1K nodes per second. Today I am seeing 20M on a simple and readily available 8 core box. 20M times faster than 1970. 200,000 times faster than the first version of Cray Blitz that had a 2250 USCF rating. Assuming a branching factor of 6 for the 1980 version of Cray Blitz, todays hardware would add 6 plies to the depth. What would 6 plies add in terms of rating? Hardware is and has been the dominant factor in computer chess strength and it will continue to be.

My category three was directed more towards better tuning, better code, better compilers, and such.

Over in the tournament forum, Lars just finished Deep Fritz 11 vs Deep Fritz 8 on equal hardware. The result: 31½-8½ a significant software gain in the last few years. Similar results can be found for all top engines.

[Uri Blass]

I have not read this whole thread, it is a large one. But I don't even see how this is arguable. If someone wants to argue it, just bring back the old machines and see how modern chess programs do on them. You use emulators but you have to throttle down their speed and memory to match what it used to be. I think everyone would see that their programs are hundreds of ELO weaker, even if the software is a couple of hundred ELO stronger. The hardware is by far going to be what caused the most gain.

I don't think Bob claimed that software hasn't advanced, only that it's not responsible for most of the gain. This is not even arguable.

I wold not be surprised if software put on ancient hardware was not even stronger (or much stronger) because things like null move and LMR require some depth to really see benefit.

I assume hash tables are not part of this discussion - I think that was a pretty big advance too and that clearly didn't become possible until hardware (memory) was there to support it. I think these other enhancement are like that, not possible or at least not very effective until the hardware was up to it. That gives us much to argue about - was it the hardware or the software?

But go back in a time machine with what you know now, and I would bet that you could not write a significantly better program. Better, perhaps, but nothing you couldn't get by waiting a couple of years for computers to double in speed.

Well, the thing is that we aren't talking about hardware vs software here. The way this discussion has been going we have gone three categories:
1. Hardware
2. Software techniques
3. all other software improvements.

When it is limited to the first two as Bob is doing, it is clear that hardware is the dominating factor. What isn't so clear is whether or not 2 & 3 combined keeps up with 1. It's not uncommon to see a top level program get updated on the same hardware with a significant gain in strength.

I don't agree with the category definitions. I would use:

(1) hardware
(2) software
(3) software features only possible because of hardware improvements

(1) is clear in what it means.

(2) is also clear in what it means.

(3) is the interesting case. For example, null move R=3 would fail miserably on a pentium-60mhz program because the depth would be so shallow, null-move errors would be rampant.

I disagree here.
I remember that
null move R=3 was proved to be productive in blitz on fast hardware based on your results and it means that it can also be productive at tournament time control on Pentium 60 mhz.

I can add that I do not compare with years like 1970 but only interested in comparison with 1993-1994 or later years.

Uri

[Don]

I don't agree with the category definitions. I would use:

(1) hardware
(2) software
(3) software features only possible because of hardware improvements

(1) is clear in what it means.

(2) is also clear in what it means.

(3) is the interesting case. For example, null move R=3 would fail miserably on a pentium-60mhz program because the depth would be so shallow, null-move errors would be rampant. But this category is _still_ a direct result of hardware improvements. The first version of "blitz" searched around 1 node per second in 1970. By 1980 on a Cray, we were doing 1K nodes per second. Today I am seeing 20M on a simple and readily available 8 core box. 20M times faster than 1970. 200,000 times faster than the first version of Cray Blitz that had a 2250 USCF rating. Assuming a branching factor of 6 for the 1980 version of Cray Blitz, todays hardware would add 6 plies to the depth. What would 6 plies add in terms of rating? Hardware is and has been the dominant factor in computer chess strength and it will continue to be.

I have been hanging out in the computer go forum for a few years now and until quite recently many believed hardware to be almost irrelevant - they were used to the model of a program that played moves right away no matter what it was running on. I had a lot of arguments about that. Of course a lot of those programs did not have anything very scalable in them, no search or just little situational searches to quickly check if a group of stones were dead. So many programs ran exactly the same no matter what you ran them on.

But what I'm primarily observing was that there was a real reluctance to admit that CPU power could be utilized in any meaningful way and that it was only about the software. After more probing, I found out that even the non-scalable programs were being updated year after year in such a way that they would not run on old hardware - so in some sense the programs were being artificially scaled to hardware. Some authors for instance admitted that their programs would take "too long" to play on older hardware.

I see a little bit of that in this discussion - although not nearly to the degree I saw in the computer go forum. For some perhaps psychological reason there is reluctance to admit that by far the vast majority of the improvement is hardware. Maybe it's just our short memories, it's real easy to forgot how horribly slow (by todays standards) the old hardware was. I expect that in 10 or 15 years we will remember the software improvements more than the hardware improvement. I remember working on an old 8 bit Z80 computer and being impressed with it's speed - it loaded programs fast and ran them fast - at least that is how my memory of it was.

[Don]

My category three was directed more towards better tuning, better code, better compilers, and such.

Over in the tournament forum, Lars just finished Deep Fritz 11 vs Deep Fritz 8 on equal hardware. The result: 31½-8½ a significant software gain in the last few years. Similar results can be found for all top engines.

How well would Deep Fritze 11 do on the OLD hardware? Although the current test is a good attempt to quantify progress, testing with the OLD hardware is a better test in my opinion because it might reveal what could have been possible WITHOUT the extra hardware. On modern hardware you are still only testing how well a modern program designed to run modern hardware does. If you intention is to refute what Bob says, this is not a very good way to do it.

Lets set up a test to see which one of us can run faster. Since you don't know how to drive, you have to run on foot, but since I can drive I get to use my car. Ok, a silly example perhaps, but just to make a point.

[Uri Blass]

I don't agree with the category definitions. I would use:

(1) hardware
(2) software
(3) software features only possible because of hardware improvements

(1) is clear in what it means.

(2) is also clear in what it means.

(3) is the interesting case. For example, null move R=3 would fail miserably on a pentium-60mhz program because the depth would be so shallow, null-move errors would be rampant. But this category is _still_ a direct result of hardware improvements. The first version of "blitz" searched around 1 node per second in 1970. By 1980 on a Cray, we were doing 1K nodes per second. Today I am seeing 20M on a simple and readily available 8 core box. 20M times faster than 1970. 200,000 times faster than the first version of Cray Blitz that had a 2250 USCF rating. Assuming a branching factor of 6 for the 1980 version of Cray Blitz, todays hardware would add 6 plies to the depth. What would 6 plies add in terms of rating? Hardware is and has been the dominant factor in computer chess strength and it will continue to be.

I have been hanging out in the computer go forum for a few years now and until quite recently many believed hardware to be almost irrelevant - they were used to the model of a program that played moves right away no matter what it was running on. I had a lot of arguments about that. Of course a lot of those programs did not have anything very scalable in them, no search or just little situational searches to quickly check if a group of stones were dead. So many programs ran exactly the same no matter what you ran them on.

But what I'm primarily observing was that there was a real reluctance to admit that CPU power could be utilized in any meaningful way and that it was only about the software. After more probing, I found out that even the non-scalable programs were being updated year after year in such a way that they would not run on old hardware - so in some sense the programs were being artificially scaled to hardware. Some authors for instance admitted that their programs would take "too long" to play on older hardware.

I see a little bit of that in this discussion - although not nearly to the degree I saw in the computer go forum. For some perhaps psychological reason there is reluctance to admit that by far the vast majority of the improvement is hardware. Maybe it's just our short memories, it's real easy to forgot how horribly slow (by todays standards) the old hardware was. I expect that in 10 or 15 years we will remember the software improvements more than the hardware improvement. I remember working on an old 8 bit Z80 computer and being impressed with it's speed - it loaded programs fast and ran them fast - at least that is how my memory of it was.

I do not know much about go but it is clear that in the last years software is the dominant factor in chess and not hardware(I agree that there were years when hardware was the dominant factor(from the time of Genius3 to the time of Fritz5.32 or from the time of Shredder7.04 to the time of Shredder9).

It is easy to test it

Rybka beta was released on December 2 2005 and was the best software at that time for single processor machines

today 3 years later Rybka3 is the best software.

Give Rybka3 hardware of 2005 and it is is going to beat every program of 2005 with hardware of today including rybka1 beta.

Uri

[bob]

I have not read this whole thread, it is a large one. But I don't even see how this is arguable. If someone wants to argue it, just bring back the old machines and see how modern chess programs do on them. You use emulators but you have to throttle down their speed and memory to match what it used to be. I think everyone would see that their programs are hundreds of ELO weaker, even if the software is a couple of hundred ELO stronger. The hardware is by far going to be what caused the most gain.

I don't think Bob claimed that software hasn't advanced, only that it's not responsible for most of the gain. This is not even arguable.

I wold not be surprised if software put on ancient hardware was not even stronger (or much stronger) because things like null move and LMR require some depth to really see benefit.

I assume hash tables are not part of this discussion - I think that was a pretty big advance too and that clearly didn't become possible until hardware (memory) was there to support it. I think these other enhancement are like that, not possible or at least not very effective until the hardware was up to it. That gives us much to argue about - was it the hardware or the software?

But go back in a time machine with what you know now, and I would bet that you could not write a significantly better program. Better, perhaps, but nothing you couldn't get by waiting a couple of years for computers to double in speed.

Well, the thing is that we aren't talking about hardware vs software here. The way this discussion has been going we have gone three categories:
1. Hardware
2. Software techniques
3. all other software improvements.

When it is limited to the first two as Bob is doing, it is clear that hardware is the dominating factor. What isn't so clear is whether or not 2 & 3 combined keeps up with 1. It's not uncommon to see a top level program get updated on the same hardware with a significant gain in strength.

I don't agree with the category definitions. I would use:

(1) hardware
(2) software
(3) software features only possible because of hardware improvements

(1) is clear in what it means.

(2) is also clear in what it means.

(3) is the interesting case. For example, null move R=3 would fail miserably on a pentium-60mhz program because the depth would be so shallow, null-move errors would be rampant.

I disagree here.
I remember that
null move R=3 was proved to be productive in blitz on fast hardware based on your results and it means that it can also be productive at tournament time control on Pentium 60 mhz.

I can add that I do not compare with years like 1970 but only interested in comparison with 1993-1994 or later years.

Uri

You are missing the point. At some point, it becomes useless at _any_ time control if the hardware is slow enough.

[Don]

I do not know much about go but it is clear that in the last years software is the dominant factor in chess and not hardware(I agree that there were years when hardware was the dominant factor(from the time of Genius3 to the time of Fritz5.32 or from the time of Shredder7.04 to the time of Shredder9).

It is easy to test it

Rybka beta was released on December 2 2005 and was the best software at that time for single processor machines

today 3 years later Rybka3 is the best software.

Give Rybka3 hardware of 2005 and it is is going to beat every program of 2005 with hardware of today including rybka1 beta.

Uri

If you limit your perspective enough, of course, software will look good.

There will be years where due to some innovation software will look like it's out-pacing hardware, but this will never be sustainable. The Rybka breakthrough is an unusual event when looked at with more than just a few years of historical perspective. Over the years we did get relatively nice little software jumps.

But even this is probably pretty minor. The test you should be doing is to see how Rybka 3 on 2005 Hardware does against the old Rybka on todays hardware. I think you will find the match is at least pretty close and that Rybka 3 barely matched the hardware improvement (and as I mentioned, Rybka's improvement is an unusual event.)

You could do this with any program. Go back 10 years and run todays programs on yesterdays computer and I'll bet yesterdays programs on todays computers beat todays programs on yesterdays computers.

How do you think a 10 year old Fritz would do on todays computers vs Rybka 3 on late 1990's hardware?

If you are right, then Rybka will still win because the software has overcome 10 years of hardware improvements. It's probably a little fairer to run modern Fritz vs Old Fritz this way, but I'm willing to give you this advantage and I will bet on Fritz to win this one.

[bob]

I don't agree with the category definitions. I would use:

(1) hardware
(2) software
(3) software features only possible because of hardware improvements

(1) is clear in what it means.

(2) is also clear in what it means.

(3) is the interesting case. For example, null move R=3 would fail miserably on a pentium-60mhz program because the depth would be so shallow, null-move errors would be rampant. But this category is _still_ a direct result of hardware improvements. The first version of "blitz" searched around 1 node per second in 1970. By 1980 on a Cray, we were doing 1K nodes per second. Today I am seeing 20M on a simple and readily available 8 core box. 20M times faster than 1970. 200,000 times faster than the first version of Cray Blitz that had a 2250 USCF rating. Assuming a branching factor of 6 for the 1980 version of Cray Blitz, todays hardware would add 6 plies to the depth. What would 6 plies add in terms of rating? Hardware is and has been the dominant factor in computer chess strength and it will continue to be.

I have been hanging out in the computer go forum for a few years now and until quite recently many believed hardware to be almost irrelevant - they were used to the model of a program that played moves right away no matter what it was running on. I had a lot of arguments about that. Of course a lot of those programs did not have anything very scalable in them, no search or just little situational searches to quickly check if a group of stones were dead. So many programs ran exactly the same no matter what you ran them on.

But what I'm primarily observing was that there was a real reluctance to admit that CPU power could be utilized in any meaningful way and that it was only about the software. After more probing, I found out that even the non-scalable programs were being updated year after year in such a way that they would not run on old hardware - so in some sense the programs were being artificially scaled to hardware. Some authors for instance admitted that their programs would take "too long" to play on older hardware.

I see a little bit of that in this discussion - although not nearly to the degree I saw in the computer go forum. For some perhaps psychological reason there is reluctance to admit that by far the vast majority of the improvement is hardware. Maybe it's just our short memories, it's real easy to forgot how horribly slow (by todays standards) the old hardware was. I expect that in 10 or 15 years we will remember the software improvements more than the hardware improvement. I remember working on an old 8 bit Z80 computer and being impressed with it's speed - it loaded programs fast and ran them fast - at least that is how my memory of it was.

I do not know much about go but it is clear that in the last years software is the dominant factor in chess and not hardware(I agree that there were years when hardware was the dominant factor(from the time of Genius3 to the time of Fritz5.32 or from the time of Shredder7.04 to the time of Shredder9).

"it is clear..." to WHO? Not to me.

It is easy to test it

Rybka beta was released on December 2 2005 and was the best software at that time for single processor machines

today 3 years later Rybka3 is the best software.

Give Rybka3 hardware of 2005 and it is is going to beat every program of 2005 with hardware of today including rybka1 beta.

Uri

The interesting point would be "what is R3's rating on today's hardware, playing 2005 programs on todays hardware?" Then "what is R3's rating on 2005 hardware playing 2005 programs on todays hardware?" You might be more than a little surprised. I'd bet the difference added by the faster hardware is more significant than the software, although 2005 is not a particularly good test since hardware has not really jumped by leaps and bounds over the last 3-4 years. Go back to 2000 and it will _really_ get interesting.

[bob]

While at the 2008 ACCA Pan American Computer Chess Championships,
Bob claimed he didn't believe software played a serious role in all the
rating improvements we've seen. He thought hardware deserved the
credit (assuming I understood the statement correctly. We were jumping
across several subjects and back that night.).

I beleive software has had much to do with it for several reasons.
I will start with one. The EBF with only MiniMax is 40. With Alpha-Beta
pruning, it drops to 6. In the early 1990's, the EBF was 4. Now, it is 2.

Dropping the EBF from 2 to 4 is huge. Lets look at a 20 ply search.
The speedup of EBF=2 vs EBF=4 is:
4^20/2^20 = 2^20 = 1,048,576

So, that is over a 1 million x speed up. Has hardware produced that much
since 1992?

Also, I believe eval improvements have caused an improvement in
rating scores.

An example of nonhardware improvements is on the SSDF rating list.
Rybka 1.0 beta score 2775 on a 450 MHz AMD.

Branching factor proves nothing because programs that do more pruning play weaker at fixed depth but I can say not based on branching factor that the improvement in software in the last years is very big and bigger than the improvement in hardware(not sure about improvement since 1992 because it is not clear how we define it but sure about improvement from 2005 to 2008).

Note that the tests of Bob can show only that hardware helped more than software for Crafty.

Tests of the SSDF showed the following results.

Rybka 3 A1200 - Deep Shredder 11 Q6600 20-19
Rybka 3 A1200 -Zappa Mexico II Q6600 20-20


A1200 = 1 x 1.2 GHz
Q6600 = 4 x 2.4 GHz

Note that both Zappa and Shredder are clearly stronger than Fruit that was the leading program in 2005 for single processor machines

I think that we can safely say that the software improvement in the last 3 years were more than 10:1 and I do not see hardware improvement of 10:1 in the last 3 years.

Uri

Can we stop with the amateurish comparisons/ Why pick different programs to compare. We had better and worse programs in 1995 as well. The question was, what have the software advances actually produced.

Feel free to name significant ones. Most would put null-move at the top, and LMR right behind it. Then we could factor in razoring, futility and extended futility from Heinz, but I already know those are very small improvements from testing by turning each off a month back or so.

what else is _significant_? Evaluation is not so interesting. I was doing passed pawn races in the 1970's. Outside passed pawns in 1995 in Crafty. So what _new_ thing since 1995 is such a big contributor? No hand-waving, no talking about ideas that _might_ have been developed. Actual documented techniques...

I have not read this whole thread, it is a large one. But I don't even see how this is arguable. If someone wants to argue it, just bring back the old machines and see how modern chess programs do on them. You use emulators but you have to throttle down their speed and memory to match what it used to be. I think everyone would see that their programs are hundreds of ELO weaker, even if the software is a couple of hundred ELO stronger. The hardware is by far going to be what caused the most gain.

I don't think Bob claimed that software hasn't advanced, only that it's not responsible for most of the gain. This is not even arguable.

I wold not be surprised if software put on ancient hardware was not even stronger (or much stronger) because things like null move and LMR require some depth to really see benefit.

I assume hash tables are not part of this discussion - I think that was a pretty big advance too and that clearly didn't become possible until hardware (memory) was there to support it. I think these other enhancement are like that, not possible or at least not very effective until the hardware was up to it. That gives us much to argue about - was it the hardware or the software?

But go back in a time machine with what you know now, and I would bet that you could not write a significantly better program. Better, perhaps, but nothing you couldn't get by waiting a couple of years for computers to double in speed.

null move and LMR work at all time controls and not only at tournament time control so I am sure that they could work at least in 120/40 with ancient hardware.

Improvement in software is both by better search and better evaluation
and I think that software is responsible for most of the gain in this century

Rybka3 can clearly use hardware of 2000 so it is easy to test it.

Give Rybka3 hardware of 2000 and give program of your choice hardware of today(but the software need to be from 2000 or earlier)

play a match and see who is going to win.

Uri

What does that prove? You want to just change one thing at a time, if you expect to measure and draw conclusions.

Best bet would be to take the best program of 2000, that is still available in updated form today and run this test:

best of 2000 on 2000 hardware, against program X (you choose here, a current program) on 2000 hardware. Now give one program 2009 hardware and re-test. Then give the other program 2009 hardware and test. I believe the 2000 program on today's hardware will be far stronger than you believe. And I believe that today's program on 2000 hardware will be far weaker than you believe.

[Don]

What does that prove? You want to just change one thing at a time, if you expect to measure and draw conclusions.

Best bet would be to take the best program of 2000, that is still available in updated form today and run this test:

best of 2000 on 2000 hardware, against program X (you choose here, a current program) on 2000 hardware. Now give one program 2009 hardware and re-test. Then give the other program 2009 hardware and test. I believe the 2000 program on today's hardware will be far stronger than you believe. And I believe that today's program on 2000 hardware will be far weaker than you believe.

Yes, that is the right test to do.

When you hear statements singing Rybka's praises (which I think is well deserved) nobody is really making he connection that they mean "Rybka on cutting edge hardware." Subconsciously we talk about Rybka as if she is the whole computing entity - devoid of hardware. But her reputation as a Grandmaster killer, etc is tied very much to the hardware she runs on.