Hardware vs Software

[Uri Blass]

First let's settle on a 10 year hardware period. The q6600 is two years old. If you want to use that as a basis, we need to return to early 1997 to choose the older hardware. The Pentium 2 (Klamath) came out around the middle of 1997, which probably means the best was the Pentium pro 200. I suspect we are _still_ talking about 200:1

This is not about simple clock frequency improvements, more modern architectures are faster for other reasons such as better speculative execution, more pipelines, register renaming, etc...

Correct me if I'm wrong, but in moving to a time handicap you seem to be ignoring the parallel search inefficiency we were both just explaining to Louis Zulli. Shouldn't that be taken into account?

None of this will matter unless it's really a close match - so I would be prepared to simple test single processor Rybka vs whatever and see what happens. If Rybka loses we have a "beta cut-off" and can stop, otherwise we must test something a little more fair and raise alpha.

If the parallel search overhead means that the ratio should really be, say, 150:1 then I don't think Rybka losing really proves your point. If there should be such a reduction, and how large it should be, is a question I am asking.

So if Rybka loses with say a 32 to 1 handicap you are saying that we should give her even less time to see if she still loses?

This is going around in circles. It is easy to quantify the hardware. I'd suggest taking the best of today, the intel I7 (core-3) and the best of late 1998. Limit it to a single chip for simplicity, but no limit on how many cores per chip. I believe this is going to be about a 200:1 time handicap to emulate the difference between the 4-core core-3 from intel and the best of 1998, which was the PII/300 processor.

For comparison, crafty on a quad-core I7 runs at 20M nodes per second, while on the single-cpu PII/300 was running at not quite 100K nodes per second. A clean and simple factor of 200x faster hardware over that period (and again, those quoting moore's law are quoting it incorrectly, it does _not_ say processor speed doubles every 2 years, it says _density_ doubles every 2 years, which is a different thing entirely). Clock speeds have gone steadily upward, but internal processor design has improved even more. Just compare a 2.0ghz core2 cpu against a 4.0ghz older processor to see what I mean.)

so that fixes the speed differential over the past ten years with high accuracy. Forget the discussions about 50:1 or the stuff about 200:1 being too high. As Bill Clinton would say, "It is what it is." And what it is is 200x.

That is almost 8 doublings, which is in the range of +600 Elo. That is going to be a great "equalizer" in this comparison. 200x is a daunting advantage to overcome. And if someone really thinks software has produced that kind of improvement, we need to test it and put it to rest once and for all...

I will accept that a program today running on 4 cores will see some overhead due to the parallel search. But I don't think it is worth arguing about whether we should scale back the speed because of the overhead. That is simply a software issue as well, as it is theoretically possible to have very little overhead. If the software can't quite use the computing power available, that is a software problem, not a hardware limit.

Then you have to accept that Fritz 5 is 622 Elo points below Rybka in current hardware. That is a bit more than the 600 points you estimate harwdare provided in 10 years.

Miguel

I don't accept that at all. That's why I suggested we run a test rather than using ratings that are very old and out of date. how many games has fritz 5.32 played _recently_ on the rating lists? That makes a huge difference and it might be better now since it is still going to beat the top programs on occasion, and with them so much higher its rating would likely drag up as well.

What do you find so wrong about these data?

Simple. Fritz 5.32 played on the SSDF 10 years ago. How many games has it played in the last 5 years? As many in the past 2-3 years as Rybka? If not, its rating will be depressed, since the old ratings tend to be static and the newer programs are advancing onward thanks to both hardware and software improvements. You can no more compare 1998 Fritz rating to 2008 Rybka rating than you can compare Capablanca's rating to Kasparov's to decide who would be better... Kasparov is not _that_ much better than Capa... Yet in Capa's time 2800+ never happened.

The comparison here between rybka and Fritz is not based on the SSDF
that did not test both with the same hardware but based on the CCRL.

The comparison is of CCRL rating of rybka and Fritz at 40/4 time control so all the games from the last 5 years.

Fritz5.32 played 2132 games for the CCRL rating list and got CCRL rating of 2642

Rybka3 64 bits single processor played 4929 games for the CCRL rating list and got rating of 3177 on the same hardware.

In other words when the hardware is the same with external small book we have
Rybka3 3177
Fritz5.32 2642

535 elo difference.

Uri

[michiguel]

First let's settle on a 10 year hardware period. The q6600 is two years old. If you want to use that as a basis, we need to return to early 1997 to choose the older hardware. The Pentium 2 (Klamath) came out around the middle of 1997, which probably means the best was the Pentium pro 200. I suspect we are _still_ talking about 200:1

This is not about simple clock frequency improvements, more modern architectures are faster for other reasons such as better speculative execution, more pipelines, register renaming, etc...

Correct me if I'm wrong, but in moving to a time handicap you seem to be ignoring the parallel search inefficiency we were both just explaining to Louis Zulli. Shouldn't that be taken into account?

None of this will matter unless it's really a close match - so I would be prepared to simple test single processor Rybka vs whatever and see what happens. If Rybka loses we have a "beta cut-off" and can stop, otherwise we must test something a little more fair and raise alpha.

If the parallel search overhead means that the ratio should really be, say, 150:1 then I don't think Rybka losing really proves your point. If there should be such a reduction, and how large it should be, is a question I am asking.

So if Rybka loses with say a 32 to 1 handicap you are saying that we should give her even less time to see if she still loses?

This is going around in circles. It is easy to quantify the hardware. I'd suggest taking the best of today, the intel I7 (core-3) and the best of late 1998. Limit it to a single chip for simplicity, but no limit on how many cores per chip. I believe this is going to be about a 200:1 time handicap to emulate the difference between the 4-core core-3 from intel and the best of 1998, which was the PII/300 processor.

For comparison, crafty on a quad-core I7 runs at 20M nodes per second, while on the single-cpu PII/300 was running at not quite 100K nodes per second. A clean and simple factor of 200x faster hardware over that period (and again, those quoting moore's law are quoting it incorrectly, it does _not_ say processor speed doubles every 2 years, it says _density_ doubles every 2 years, which is a different thing entirely). Clock speeds have gone steadily upward, but internal processor design has improved even more. Just compare a 2.0ghz core2 cpu against a 4.0ghz older processor to see what I mean.)

so that fixes the speed differential over the past ten years with high accuracy. Forget the discussions about 50:1 or the stuff about 200:1 being too high. As Bill Clinton would say, "It is what it is." And what it is is 200x.

That is almost 8 doublings, which is in the range of +600 Elo. That is going to be a great "equalizer" in this comparison. 200x is a daunting advantage to overcome. And if someone really thinks software has produced that kind of improvement, we need to test it and put it to rest once and for all...

I will accept that a program today running on 4 cores will see some overhead due to the parallel search. But I don't think it is worth arguing about whether we should scale back the speed because of the overhead. That is simply a software issue as well, as it is theoretically possible to have very little overhead. If the software can't quite use the computing power available, that is a software problem, not a hardware limit.

Then you have to accept that Fritz 5 is 622 Elo points below Rybka in current hardware. That is a bit more than the 600 points you estimate harwdare provided in 10 years.

Miguel

I don't accept that at all. That's why I suggested we run a test rather than using ratings that are very old and out of date. how many games has fritz 5.32 played _recently_ on the rating lists? That makes a huge difference and it might be better now since it is still going to beat the top programs on occasion, and with them so much higher its rating would likely drag up as well.

What do you find so wrong about these data?

Simple. Fritz 5.32 played on the SSDF 10 years ago. How many games has it played in the last 5 years? As many in the past 2-3 years as Rybka? If not, its rating will be depressed, since the old ratings tend to be static and the newer programs are advancing onward thanks to both hardware and software improvements. You can no more compare 1998 Fritz rating to 2008 Rybka rating than you can compare Capablanca's rating to Kasparov's to decide who would be better... Kasparov is not _that_ much better than Capa... Yet in Capa's time 2800+ never happened.

That was my second attempt to introduce these data. It was the third time if we count Uri mentioned it before. Could you please read it? I took the time to go to the website, search it, and paste it here, to save you time. The answers are there already:

Code: Select all

Tested by CCRL team, 2005-2009, http://computerchess.org.uk/ccrl/404/

I see nothing wrong with the data.

Miguel

Code: Select all

CCRL 40/4 Rating List - Custom engine selection
388092 games played by 744 programs, run by 12 testers
Ponder off, General books (up to 12 moves), 3-4-5 piece EGTB
Time control: Equivalent to 40 moves in 4 minutes on Athlon 64 X2 4600+ (2.4 GHz)
Computed on January 10, 2009 with Bayeselo based on 388'092 games
Tested by CCRL team, 2005-2009, http://computerchess.org.uk/ccrl/404/

Rank                 Engine                   ELO   +    -   Score  AvOp  Games
1                   Fritz 5.32              2642  +13  -13  53.2%  -24.5  2132

Miguel

So let's run the test rather than speculating...

I have some Crafty versions that should be right for that time frame. Crafty 15.0 was the first parallel search version. I suspect something in the 16.x versions or possibly 17.x versions was used at the end of 1998. Crafty ran on a quad pentium pro early in 1998 whe
n version 15.0 was done...

[Dirt]

According to this the Pentium II/450 was introduced at the end of August 2008, so I expect some computers with the chip were available by the end of the year.

Based on my recollections, a Pentium/300 (not PII) should run Crafty at about 75M n/s. I think that's in rough agreement with Bob's claim of 100M n/s on a PII/300.

You mean August 1998. And 75K n/s. I remember, on my P200 Crafty was about 75K n/s, on PII 450 it could be 150K n/s. I think Uri's factor of 75 in 10 years is the best estimate. This, if taking into account the latest high-end boxes. But Rybka is able to beat Fritz 5 in 1:200 time controls. Ok, Bob tried to explain that hardware improvement leads to software improvement, but besides 64-bit, it is still software tuning to faster hardware.

Kai

I ran on a PII/400 quad for a couple of years. I was getting about 100K. That was on a PII/400 xeon with 2mb of L2 cache, which was faster than the usual PII/450 single-cpu box that came in a desktop. The quad had 4-way memory interleaving to further improve memory bandwidth. I didn't get the quad until the middle of 1999 however. If you prefer to use that particular processor, then 100K is a real good estimate. If you want to take that up to 125K that's OK. I know that on today's latest, as of December 2008, I could hit 20M on a single chip quad-core. I get a factor of 160X faster. If you want to account for 4-processor search overhead, using the 3.3x speedup numbers from a long quad-cpu test I discussed and which we analyzed here on CCC, then we change that to 132:1...

Is everyone happy with that???

There are some benchmarks of Crafty 17.06 on (among others) Pentium IIs here.
At the bottom of the page the method is described, and they don't seem to use the "bench" command, so I don't know if these results have any meaning for this discussion.

[Laskos]

According to this the Pentium II/450 was introduced at the end of August 2008, so I expect some computers with the chip were available by the end of the year.

Based on my recollections, a Pentium/300 (not PII) should run Crafty at about 75M n/s. I think that's in rough agreement with Bob's claim of 100M n/s on a PII/300.

You mean August 1998. And 75K n/s. I remember, on my P200 Crafty was about 75K n/s, on PII 450 it could be 150K n/s. I think Uri's factor of 75 in 10 years is the best estimate. This, if taking into account the latest high-end boxes. But Rybka is able to beat Fritz 5 in 1:200 time controls. Ok, Bob tried to explain that hardware improvement leads to software improvement, but besides 64-bit, it is still software tuning to faster hardware.

Kai

I ran on a PII/400 quad for a couple of years. I was getting about 100K. That was on a PII/400 xeon with 2mb of L2 cache, which was faster than the usual PII/450 single-cpu box that came in a desktop. The quad had 4-way memory interleaving to further improve memory bandwidth. I didn't get the quad until the middle of 1999 however. If you prefer to use that particular processor, then 100K is a real good estimate. If you want to take that up to 125K that's OK. I know that on today's latest, as of December 2008, I could hit 20M on a single chip quad-core. I get a factor of 160X faster. If you want to account for 4-processor search overhead, using the 3.3x speedup numbers from a long quad-cpu test I discussed and which we analyzed here on CCC, then we change that to 132:1...

Is everyone happy with that???

There are some benchmarks of Crafty 17.06 on (among others) Pentium IIs here.
At the bottom of the page the method is described, and they don't seem to use the "bench" command, so I don't know if these results have any meaning for this discussion.

The benchmark is good. It shows that the P200 MMX is at 72K n/s, I remember 75K n/s on my P200 MMX. At the end of 1998 there were available 400 MHz boxes with each processor giving 200K n/s. It shows that Bob's quad PII 400 Xeon would give 625K n/s (it is explicit there). I still think that Uri's factor of 75 is the best hardware estimate (it can be even lowered), while the software increased by a factor of 200-300 (in Rybka's case).

Kai

[bob]

According to this the Pentium II/450 was introduced at the end of August 2008, so I expect some computers with the chip were available by the end of the year.

Based on my recollections, a Pentium/300 (not PII) should run Crafty at about 75M n/s. I think that's in rough agreement with Bob's claim of 100M n/s on a PII/300.

You mean August 1998. And 75K n/s. I remember, on my P200 Crafty was about 75K n/s, on PII 450 it could be 150K n/s. I think Uri's factor of 75 in 10 years is the best estimate. This, if taking into account the latest high-end boxes. But Rybka is able to beat Fritz 5 in 1:200 time controls. Ok, Bob tried to explain that hardware improvement leads to software improvement, but besides 64-bit, it is still software tuning to faster hardware.

Kai

I ran on a PII/400 quad for a couple of years. I was getting about 100K. That was on a PII/400 xeon with 2mb of L2 cache, which was faster than the usual PII/450 single-cpu box that came in a desktop. The quad had 4-way memory interleaving to further improve memory bandwidth. I didn't get the quad until the middle of 1999 however. If you prefer to use that particular processor, then 100K is a real good estimate. If you want to take that up to 125K that's OK. I know that on today's latest, as of December 2008, I could hit 20M on a single chip quad-core. I get a factor of 160X faster. If you want to account for 4-processor search overhead, using the 3.3x speedup numbers from a long quad-cpu test I discussed and which we analyzed here on CCC, then we change that to 132:1...

Is everyone happy with that???

There are some benchmarks of Crafty 17.06 on (among others) Pentium IIs here.
At the bottom of the page the method is described, and they don't seem to use the "bench" command, so I don't know if these results have any meaning for this discussion.

The benchmark is good. It shows that the P200 MMX is at 72K n/s, I remember 75K n/s on my P200 MMX. At the end of 1998 there were available 400 MHz boxes with each processor giving 200K n/s. It shows that Bob's quad PII 400 Xeon would give 625K n/s (it is explicit there). I still think that Uri's factor of 75 is the best hardware estimate (it can be even lowered), while the software increased by a factor of 200-300 (in Rybka's case).

Kai

Couple of points. First, no way you could get 75K on a P200/mmx. The pentium pro 200 would do 75 K, but the basic pentium, all the way thru the P200/mmx was a different processor architecture, no out of order execution or any of that. I had a p200/mmx laptop that would not break 50K. The pentium pro 200 was getting around 75K. I ran on it in Jakarta, and in my office, until I got the quad pentium pro 200 box.

If you take the 400mhz processor, which I am not sure was around in 1988, at least the xeon 2mb L2 that I had, then 150K nodes per second would be the answer. Late in 2008 an i7 was producing 20M nodes per second.. If you divide 20M by 150K you get a raw factor of 133x. If you factor in the 3.3/4.0 speedup for a parallel search you get a factor of 110x.

We could settle on 100:1 for simplicity, although that is _still_ a daunting advantage. I find it amusing that so many are convinced that software is most of the improvement, yet we are quibbling over a 10%-20% speed difference in hardware. It seems, to me, that perhaps most really do believe that hardware is the more important contributor to today's ratings in spite of rhetoric to the contrary...

Once we get to the improvement in hardware as settled, then there are going to be enough arguments to make the test hopeless anyway. 1 sec to 100 secs is a _long_ game. At 60 moves as a reasonable average game length, that turns into almost 2 hours per game. Even at 256 games at a time on my cluster, it takes a long time to play enough games to get good answers. And some are not going to accept 1 second vs 100 seconds. And want 1 minute to 100 minutes, which is not going to happen with me...

[BubbaTough]

Given that one area of steady progress in software improvements has been in decreasing the branching factor, it would not be surprising if the longer the game, the more valuable the software changes prove. Perhaps, no matter what hardware difference is claimed, at some length of time, perhaps days or weeks, software changes prove more important than hardware. Conversely, at some short length of time, the speed of hardware is much more important than software changes.

-Sam

[bob]

Given that one area of steady progress in software improvements has been in decreasing the branching factor, it would not be surprising if the longer the game, the more valuable the software changes prove. Perhaps, no matter what hardware difference is claimed, at some length of time, perhaps days or weeks, software changes prove more important than hardware. Conversely, at some short length of time, the speed of hardware is much more important than software changes.

-Sam

That is based on conjecture rather than fact. For example, I have run extensive tests on null-move, and could find no difference in Elo gain/loss for several time controls. And turning it off is not a huge deal. Same thing for LMR. Your assumption is that today's plies are the same as plies 10 years ago, we just get more of them with more and more time because of the reduced branching factor. That is not a given, in my opinion. Which means that the 'additional plies" with longer time controls is not necessarily _that_ much better overall.

You can probably find the posts where I ran some tests with null move or LMR or both disabled... But your argument is flawed in a basic way. What you are really saying is that the _faster_ the hardware gets, the better these algorithms perform. And that is not the point most want to accept.

Again, hardware has brought more gain than software. How much is TBD, but I'd bet 2/3 for hardware, 1/3 for software myself. And anyone that says we need a longer time control to test is only making the point more clear.

[Dirt]

Couple of points. First, no way you could get 75K on a P200/mmx. The Pentium Pro 200 would do 75 K, but the basic Pentium, all the way thru the P200/mmx was a different processor architecture, no out of order execution or any of that. I had a p200/mmx laptop that would not break 50K. The Pentium pro 200 was getting around 75K. I ran on it in Jakarta, and in my office, until I got the quad Pentium pro 200 box.

If you take the 400mhz processor, which I am not sure was around in 1988, at least the xeon 2mb L2 that I had, then 150K nodes per second would be the answer. Late in 2008 an i7 was producing 20M nodes per second.. If you divide 20M by 150K you get a raw factor of 133x. If you factor in the 3.3/4.0 speedup for a parallel search you get a factor of 110x.

We could settle on 100:1 for simplicity, although that is _still_ a daunting advantage. I find it amusing that so many are convinced that software is most of the improvement, yet we are quibbling over a 10%-20% speed difference in hardware. It seems, to me, that perhaps most really do believe that hardware is the more important contributor to today's ratings in spite of rhetoric to the contrary...

Once we get to the improvement in hardware as settled, then there are going to be enough arguments to make the test hopeless anyway. 1 sec to 100 secs is a _long_ game. At 60 moves as a reasonable average game length, that turns into almost 2 hours per game. Even at 256 games at a time on my cluster, it takes a long time to play enough games to get good answers. And some are not going to accept 1 second vs 100 seconds. And want 1 minute to 100 minutes, which is not going to happen with me...

One hundred to one does sound like a nice number. You seem convinced that is too little, others are sure it's too much.

Are you considering 1 vs 100 seconds as an increment? That would be unfair to Fritz. The fairest time would have the time they were tuned for as the geometric mean. While everyone I'm sure tries to tune for longer time controls, in practice I think it must be much shorter, perhaps someplace around 30 seconds per game. I think either 2"/40 vs 200"/40 or 1" + 0.03 vs 100" + 3 would be in the right ballpark, assuming Rybka's time management doesn't completely fail at that level.

[BubbaTough]

Given that one area of steady progress in software improvements has been in decreasing the branching factor, it would not be surprising if the longer the game, the more valuable the software changes prove. Perhaps, no matter what hardware difference is claimed, at some length of time, perhaps days or weeks, software changes prove more important than hardware. Conversely, at some short length of time, the speed of hardware is much more important than software changes.

-Sam

...
That is based on conjecture rather than fact. For example, I have run extensive tests on null-move, and could find no difference in Elo gain/loss for several time controls. And turning it off is not a huge deal. Same thing for LMR. Your assumption is that today's plies are the same as plies 10 years ago, we just get more of them with more and more time because of the reduced branching factor. That is not a given, in my opinion. Which means that the 'additional plies" with longer time controls is not necessarily _that_ much better overall.
...

It is not just based on conjecture, it IS conjecture. Notice key words/phrases such as "Not surprising if" and "Perhaps". I have not made any assumptions about equal plies...unequal plys were definitely on my mind as I typed (otherwise I would not need my qualifiers). My personal experience has been older programs make much worse use of extra time on today's hardware than today's programs. MUCH worse. Thus my speculation...

Though it hardly needs to be said: as always, my speculative hypothesis based on flimsy data might be wrong. I don't have a particularly strong belief that its true myself, it just sounds plausible...perhaps 63% likely.

-Sam

[Uri Blass]

Couple of points. First, no way you could get 75K on a P200/mmx. The Pentium Pro 200 would do 75 K, but the basic Pentium, all the way thru the P200/mmx was a different processor architecture, no out of order execution or any of that. I had a p200/mmx laptop that would not break 50K. The Pentium pro 200 was getting around 75K. I ran on it in Jakarta, and in my office, until I got the quad Pentium pro 200 box.

If you take the 400mhz processor, which I am not sure was around in 1988, at least the xeon 2mb L2 that I had, then 150K nodes per second would be the answer. Late in 2008 an i7 was producing 20M nodes per second.. If you divide 20M by 150K you get a raw factor of 133x. If you factor in the 3.3/4.0 speedup for a parallel search you get a factor of 110x.

We could settle on 100:1 for simplicity, although that is _still_ a daunting advantage. I find it amusing that so many are convinced that software is most of the improvement, yet we are quibbling over a 10%-20% speed difference in hardware. It seems, to me, that perhaps most really do believe that hardware is the more important contributor to today's ratings in spite of rhetoric to the contrary...

Once we get to the improvement in hardware as settled, then there are going to be enough arguments to make the test hopeless anyway. 1 sec to 100 secs is a _long_ game. At 60 moves as a reasonable average game length, that turns into almost 2 hours per game. Even at 256 games at a time on my cluster, it takes a long time to play enough games to get good answers. And some are not going to accept 1 second vs 100 seconds. And want 1 minute to 100 minutes, which is not going to happen with me...

One hundred to one does sound like a nice number. You seem convinced that is too little, others are sure it's too much.

Are you considering 1 vs 100 seconds as an increment? That would be unfair to Fritz. The fairest time would have the time they were tuned for as the geometric mean. While everyone I'm sure tries to tune for longer time controls, in practice I think it must be much shorter, perhaps someplace around 30 seconds per game. I think either 2"/40 vs 200"/40 or 1" + 0.03 vs 100" + 3 would be in the right ballpark, assuming Rybka's time management doesn't completely fail at that level.

I do not care about the time the program were tuned for.

progress of software is based on result and not based on intentions.
If program X was better at 120/40 ponder on ssdf games and weaker at blitz at 1998 then I consider program X to be better for the discussion.

120/40 ponder on games mean something equivalent to 4.5 minutes ponder off on P200MMX so it clearly mean more than 1 second on hardware of today.

Uri