Hardware vs Software

[Dirt]

How do you think a 10 year old Fritz would do on todays computers vs Rybka 3 on late 1990's hardware?

If you are right, then Rybka will still win because the software has overcome 10 years of hardware improvements. It's probably a little fairer to run modern Fritz vs Old Fritz this way, but I'm willing to give you this advantage and I will bet on Fritz to win this one.

I think it would be close, but I'd take Uri's side. Using Fritz on both sides doesn't sound fair to me at all.

[Don]

How do you think a 10 year old Fritz would do on todays computers vs Rybka 3 on late 1990's hardware?

If you are right, then Rybka will still win because the software has overcome 10 years of hardware improvements. It's probably a little fairer to run modern Fritz vs Old Fritz this way, but I'm willing to give you this advantage and I will bet on Fritz to win this one.

I think it would be close, but I'd take Uri's side. Using Fritz on both sides doesn't sound fair to me at all.

This match would not be close.

The reason I say Fritz vs Fritz is that it is a program that has been under active development the whole time and reflects most smoothly the state of the art. Rybka didn't exist back then and in AI termnology it's a bit of a local optima. A bump in the landscape which is not smooth.

If the best program is used (Rybka) then the best program of 10 years ago should also be used, whatever it is. Perhaps it is Fritz? I don't really know but I don't think it matters as even Rybka would lose with those odds against it.

[Mike S.]

I am (almost) doing this test in practice. My old computer has an AMD AthlonXP cpu which I cannot run faster than 1.5 GHz. The RAM is not fast, and usually I give engines 128 MB hash, or max. 192. Rybka is as superior there, as anywhere. It is not a year 2000 hardware but a few years later I think, anyway, I never noticed big distortions of relative engine strengths after replacing computers with new and faster ones. Also, I was always continueing to use some old engines with my new computers each, and I never noticed a drop of (relative) performance.

These are "general impressions", in other words I did not measure that, and if the discussion is about very small differences like +/- 30 Elo, than I didn't notice them.

I recall only one big effect of hardware change, with a program which perfomed MUCH better on the Athlon than on Intel P3 before (disregarding the clock rate difference): Chess System Tal II. The Athlon vs. P3 speed gain was +40%. But that is something very unique.

P.S. I usually don't post in this subforum but I lurked into it after seeing your name. I have Socrates 3.0, which I liked when it was new. Don't hurry with version 4.0 Haste makes waste.

P.P.S. In the Rybka forum, I read that you and Larry Kaufman cooperate again. This is wonderful news.

[Don]

I am (almost) doing this test in practice. My old computer has an AMD AthlonXP cpu which I cannot run faster than 1.5 GHz. The RAM is not fast, and usually I give engines 128 MB hash, or max. 192. Rybka is as superior there, as anywhere. It is not a year 2000 hardware but a few years later I think, anyway, I never noticed big distortions of relative engine strengths after replacing computers with new and faster ones. Also, I was always continueing to use some old engines with my new computers each, and I never noticed a drop of (relative) performance.

These are "general impressions", in other words I did not measure that, and if the discussion is about very small differences like +/- 30 Elo, than I didn't notice them.

I recall only one big effect of hardware change, with a program which perfomed MUCH better on the Athlon than on Intel P3 before (disregarding the clock rate difference): Chess System Tal II. The Athlon vs. P3 speed gain was +40%. But that is something very unique.

P.S. I usually don't post in this subforum but I lurked into it after seeing your name. I have Socrates 3.0, which I liked when it was new. Don't hurry with version 4.0 Haste makes waste.

P.P.S. In the Rybka forum, I read that you and Larry Kaufman cooperate again. This is wonderful news.

Rybka is going to be superior on any hardware, that's not the point. The question has to do with how superior is it? If you assume that Moores law gives a double every 2 years, then you have 5 doublings. Fritz running 32 times faster is going to win. Even the older Fritz.

But if you are going back 10 years you also have to also give Rybka a single processor computer to run on (we are talking about hardware, right?) because that was commodity hardware back then. And you must give Fritz the best of what is available today. (It's true that dual processor machines were available 10 years ago, but to be fair almost nobody had them, but lots of people have 4 core machines today.)

I don't even know if Fritz is a fair test, because it might not be the best that was available 10 years ago. Does anyone know?

You should also adjust the hash table size accordingly to what was commonly available since we are talking about hardware right? "Little details" like this are probably ignored, but that is also a hardware advance. I believe 10 years ago it was rare to have even 1/2 gig of memory in your machine.

[Uri Blass]

I do not know much about go but it is clear that in the last years software is the dominant factor in chess and not hardware(I agree that there were years when hardware was the dominant factor(from the time of Genius3 to the time of Fritz5.32 or from the time of Shredder7.04 to the time of Shredder9).

It is easy to test it

Rybka beta was released on December 2 2005 and was the best software at that time for single processor machines

today 3 years later Rybka3 is the best software.

Give Rybka3 hardware of 2005 and it is is going to beat every program of 2005 with hardware of today including rybka1 beta.

Uri

If you limit your perspective enough, of course, software will look good.

There will be years where due to some innovation software will look like it's out-pacing hardware, but this will never be sustainable. The Rybka breakthrough is an unusual event when looked at with more than just a few years of historical perspective. Over the years we did get relatively nice little software jumps.

But even this is probably pretty minor. The test you should be doing is to see how Rybka 3 on 2005 Hardware does against the old Rybka on todays hardware. I think you will find the match is at least pretty close and that Rybka 3 barely matched the hardware improvement (and as I mentioned, Rybka's improvement is an unusual event.)

You could do this with any program. Go back 10 years and run todays programs on yesterdays computer and I'll bet yesterdays programs on todays computers beat todays programs on yesterdays computers.

How do you think a 10 year old Fritz would do on todays computers vs Rybka 3 on late 1990's hardware?

If you are right, then Rybka will still win because the software has overcome 10 years of hardware improvements. It's probably a little fairer to run modern Fritz vs Old Fritz this way, but I'm willing to give you this advantage and I will bet on Fritz to win this one.

I guess that the speed improvement from hardware in the last 10 years is less than 50:1 if you do not include hash tables(and I guess that the advantage from bigger hash is probably something that is equivalent to less than 2:1 speed improvement).

If I am correct then
I expect Rybka3 to win at least if you play 120/40 time control and not blitz.

I did not test rybka3 but I tested Rybka2.3.2a against movei00.8.438 and at long time control rybka2.3.2a clearly wins with 10:1 time handicap and long time control mean 4 minutes/40 moves for rybka2.3.2a against 40 minutes/40 moves for movei(I do not remember the exact result but rybka scored more than 55 points out of 100 and rybka clearly performed better with longer time control in these handicapped matches when I did not use a lot of hash and I think that I used 64 mbytes hash for both sides when I do not believe more hash could help movei to win the match).

Movei is clearly stronger than Fritz of 1998-1999 based on rating lists.
Rybka3 is clearly stronger than Rybka2.3.2a so I expect Rybka3 to win Fritz of 1999 even with 100:1 time handicap if you use 120/40 time control(even if you give Fritz more hash to emulate hardware of today).

Uri

[Uri Blass]

I am (almost) doing this test in practice. My old computer has an AMD AthlonXP cpu which I cannot run faster than 1.5 GHz. The RAM is not fast, and usually I give engines 128 MB hash, or max. 192. Rybka is as superior there, as anywhere. It is not a year 2000 hardware but a few years later I think, anyway, I never noticed big distortions of relative engine strengths after replacing computers with new and faster ones. Also, I was always continueing to use some old engines with my new computers each, and I never noticed a drop of (relative) performance.

These are "general impressions", in other words I did not measure that, and if the discussion is about very small differences like +/- 30 Elo, than I didn't notice them.

I recall only one big effect of hardware change, with a program which perfomed MUCH better on the Athlon than on Intel P3 before (disregarding the clock rate difference): Chess System Tal II. The Athlon vs. P3 speed gain was +40%. But that is something very unique.

P.S. I usually don't post in this subforum but I lurked into it after seeing your name. I have Socrates 3.0, which I liked when it was new. Don't hurry with version 4.0 Haste makes waste.

P.P.S. In the Rybka forum, I read that you and Larry Kaufman cooperate again. This is wonderful news.

Rybka is going to be superior on any hardware, that's not the point. The question has to do with how superior is it? If you assume that Moores law gives a double every 2 years, then you have 5 doublings. Fritz running 32 times faster is going to win. Even the older Fritz.

But if you are going back 10 years you also have to also give Rybka a single processor computer to run on (we are talking about hardware, right?) because that was commodity hardware back then. And you must give Fritz the best of what is available today. (It's true that dual processor machines were available 10 years ago, but to be fair almost nobody had them, but lots of people have 4 core machines today.)

I don't even know if Fritz is a fair test, because it might not be the best that was available 10 years ago. Does anyone know?

You should also adjust the hash table size accordingly to what was commonly available since we are talking about hardware right? "Little details" like this are probably ignored, but that is also a hardware advance. I believe 10 years ago it was rare to have even 1/2 gig of memory in your machine.

You say:
"Fritz running 32 times faster is going to win. Even the older Fritz."

I am not sure about it
It is probably going to be closed match and the question who is going to win is dependent on the time control and the opening book that is used.

Fritz of today has a better opening book but even if we ignore this advantage and play matches from fixed positions it is not clear for me who is going to win.

For the question if Fritz was the best 10 years ago then
Fritz5 (or Fritz5.32) was the ssdf leader in the end of 1998 and in part of 1999
only hiarcs7 and chessmaster6000 were at similiar level at that time

http://www.geocities.com/CapeCanaveral/ ... df9808.htm

Uri

[Mike S.]

I am going to run a long test match between Fritz 5.32 and Rybka 2.3.2a, both with default engine settings and with ponder off, on single cpu AMD Athlon@1.5 and with predefined opening variations, switching sides each.

I will report back with the results on Thursday or Friday.

[Don]

I am going to run a long test match between Fritz 5.32 and Rybka 2.3.2a, both with default engine settings and with ponder off, on single cpu AMD Athlon@1.5 and with predefined opening variations, switching sides each.

I will report back with the results on Thursday or Friday.

But that will only show us that Rybka is superior. I expect Rybka to be superior on ANY hardware by approximately the same amount, so I don't see what this would prove.

[Dirt]

This match would not be close.

The reason I say Fritz vs Fritz is that it is a program that has been under active development the whole time and reflects most smoothly the state of the art. Rybka didn't exist back then and in AI terminology it's a bit of a local optima. A bump in the landscape which is not smooth.

If the best program is used (Rybka) then the best program of 10 years ago should also be used, whatever it is. Perhaps it is Fritz? I don't really know but I don't think it matters as even Rybka would lose with those odds against it.

The reason I say not Fritz is that it is not the best representative of chess software today; that is Rybka. You can't measure software advance by comparing a program that is not the most advanced. Restricting it to just one program, unless it was the best at both the start and end points, is fundamentally wrong.

Ten years ago the best program might have been Hiarcs 7.0, or possibly some version of Shredder or Fritz.

[Mike S.]

Ok; of course I would expect Rybka to win this, too. But I thought this discussion is about new software being less superior on old hardware, than on new hardware. But if nobody expects that Rybka is relatively worse on old hardware anyway, than this point is void and I don't see the sense of it.

But thanks for the remark; now I can save time and efforts by not doing a useless test.

If anyone thinks new software is worse on old hardware: It can be tested. New and old hard- and software is here, it still works, and it can be tested. That would provide facts instead of "talking"...