Crafty tests show that Software has advanced more.

[Don]

I guess I understand what you are saying, and I have always agreed with you (in fact I consider it quite obvious) that hardware advances have contributed more than software, but by using modern crafty instead of a top program you seem to be lopping off around 300 elo of today's advances (not sure what the exact number is but I suspect its pretty big).

Bob has already backed off this issue, the only thing his test proves is that Crafty has not improved as much due to software. His test proves that and he is not claiming that his test proves anything else.

And when you think about it there is no reasonable test we could have done to settle the issue. Such a test really needed to be automated and using Crafty was the only reasonable test we could construct.

It's unfortunate that this does not address what the original debate was about, but I don't have any suggestions on how to properly do this.

[rbarreira]

I guess I understand what you are saying, and I have always agreed with you (in fact I consider it quite obvious) that hardware advances have contributed more than software, but by using modern crafty instead of a top program you seem to be lopping off around 300 elo of today's advances (not sure what the exact number is but I suspect its pretty big).

Bob has already backed off this issue, the only thing his test proves is that Crafty has not improved as much due to software. His test proves that and he is not claiming that his test proves anything else.

And when you think about it there is no reasonable test we could have done to settle the issue. Such a test really needed to be automated and using Crafty was the only reasonable test we could construct.

It's unfortunate that this does not address what the original debate was about, but I don't have any suggestions on how to properly do this.

As I said in the new thread I created, we should be able to get a pretty good answer to the original question by taking into account the distance from crafty to the best chess engine both in 1995 and 2010. For me the conclusion was that software contributed almost as much as hardware in terms of elo, but probably a bit less.

[BubbaTough]

I guess I understand what you are saying, and I have always agreed with you (in fact I consider it quite obvious) that hardware advances have contributed more than software, but by using modern crafty instead of a top program you seem to be lopping off around 300 elo of today's advances (not sure what the exact number is but I suspect its pretty big).

Bob has already backed off this issue, the only thing his test proves is that Crafty has not improved as much due to software. His test proves that and he is not claiming that his test proves anything else.

And when you think about it there is no reasonable test we could have done to settle the issue. Such a test really needed to be automated and using Crafty was the only reasonable test we could construct.

It's unfortunate that this does not address what the original debate was about, but I don't have any suggestions on how to properly do this.

Well, I can imagine ways it can be done, but they would be more work than anyone would want to put in. What I think is very clear is that it is absolutely amazing how much software has improved in that time. Given the exponential advances of hardware, its easy to attribute everything to hardware. It seems though that software may have also been improving at an exponential rate, which is really astounding. The fact that hardware may be edging out software takes nothing away from the astounding rate of software improvement in my mind...particularly when you compare the money being spent on hardware improvement vs. chess software improvement over that period .

-Sam

[bob]

I guess I understand what you are saying, and I have always agreed with you (in fact I consider it quite obvious) that hardware advances have contributed more than software, but by using modern crafty instead of a top program you seem to be lopping off around 300 elo of today's advances (not sure what the exact number is but I suspect its pretty big).

Bob has already backed off this issue, the only thing his test proves is that Crafty has not improved as much due to software. His test proves that and he is not claiming that his test proves anything else.

And when you think about it there is no reasonable test we could have done to settle the issue. Such a test really needed to be automated and using Crafty was the only reasonable test we could construct.

It's unfortunate that this does not address what the original debate was about, but I don't have any suggestions on how to properly do this.

As I said in the new thread I created, we should be able to get a pretty good answer to the original question by taking into account the distance from crafty to the best chess engine both in 1995 and 2010. For me the conclusion was that software contributed almost as much as hardware in terms of elo, but probably a bit less.

The problem is the measurement. The measurements I provided are done under a carefully controlled set of circumstances. Identical hardware. Simplistic operating system without a lot of useless stuff running. Consistent set of opponents that never change at all. Consistent set of opening positions that never change. No opening books allowed. No endgame tables. No learning of any kind allowed.

Taking any rating list has a number of compromises. Old programs. Variable hardware. Different opening books. Learning on in some tests, off in others, reset periodically in others.

Rybka is the current top, But I can't run Rybka under the same controlled circumstances, myself. If we want to agree that StockFish 1.8 is close enough to call them the same, I can certainly run that test. We have been planning to update our 1.6 to 1.8 anyway. I just don't change very often because it doesn't make my test any more accurate since I am trying to catch up to them, and changing the version invalidates all the old data that we have since now one of the common opponents has changed.

If there is interest, since I have 1.8 on my cluster, I can certainly run 23.0-23.4 against it and get a pretty good indication of how far apart 23.4 and SF 1.8 is. That is about the best number I can provide.

[Don]

I guess I understand what you are saying, and I have always agreed with you (in fact I consider it quite obvious) that hardware advances have contributed more than software, but by using modern crafty instead of a top program you seem to be lopping off around 300 elo of today's advances (not sure what the exact number is but I suspect its pretty big).

Bob has already backed off this issue, the only thing his test proves is that Crafty has not improved as much due to software. His test proves that and he is not claiming that his test proves anything else.

And when you think about it there is no reasonable test we could have done to settle the issue. Such a test really needed to be automated and using Crafty was the only reasonable test we could construct.

It's unfortunate that this does not address what the original debate was about, but I don't have any suggestions on how to properly do this.

As I said in the new thread I created, we should be able to get a pretty good answer to the original question by taking into account the distance from crafty to the best chess engine both in 1995 and 2010. For me the conclusion was that software contributed almost as much as hardware in terms of elo, but probably a bit less.

Actually, I believe the contribution is about the same all things considered. Bob's test actually put that in doubt for me, it makes it seem that software is the biggest contributor when you add the Rybka factor, parallel efficiency and so on. But there are so many variables to consider that in my mind have no clear answer that I think this is useless and why I have refrained from commenting. To me the test surely "demonstrates" that at least within a year or two (in either direction) you can say that software and hardware are equal contributors. If you think you will get Bob to agree that this is the case - don't hold your breath waiting.

[bob]

I guess I understand what you are saying, and I have always agreed with you (in fact I consider it quite obvious) that hardware advances have contributed more than software, but by using modern crafty instead of a top program you seem to be lopping off around 300 elo of today's advances (not sure what the exact number is but I suspect its pretty big).

Bob has already backed off this issue, the only thing his test proves is that Crafty has not improved as much due to software. His test proves that and he is not claiming that his test proves anything else.

And when you think about it there is no reasonable test we could have done to settle the issue. Such a test really needed to be automated and using Crafty was the only reasonable test we could construct.

It's unfortunate that this does not address what the original debate was about, but I don't have any suggestions on how to properly do this.

As I said in the new thread I created, we should be able to get a pretty good answer to the original question by taking into account the distance from crafty to the best chess engine both in 1995 and 2010. For me the conclusion was that software contributed almost as much as hardware in terms of elo, but probably a bit less.

Actually, I believe the contribution is about the same all things considered. Bob's test actually put that in doubt for me, it makes it seem that software is the biggest contributor when you add the Rybka factor, parallel efficiency and so on. But there are so many variables to consider that in my mind have no clear answer that I think this is useless and why I have refrained from commenting. To me the test surely "demonstrates" that at least within a year or two (in either direction) you can say that software and hardware are equal contributors. If you think you will get Bob to agree that this is the case - don't hold your breath waiting.

If you'd only read, I believe I have already said _exactly_ that. +300 still leaves the hardware with a 140 point (or maybe more, I could not go back in time far enough to see if those 1995 doublings would be more than +80 or not. I suspect they would. I am not sure +300 is correct.

A 200 point rating advantage is still an advantage that says you will win about 3 of every 4 games played, so it is not insignificant. I'm right now working on re-calibrating all my tests using stockfish 1.8, which is real close to Rybka. That will at least give a reasonable estimate of the gap between Crafty and Rybka, and an exact measure of the gap between Crafty and Stockfish when no book or learning is allowed.

[Don]

I guess I understand what you are saying, and I have always agreed with you (in fact I consider it quite obvious) that hardware advances have contributed more than software, but by using modern crafty instead of a top program you seem to be lopping off around 300 elo of today's advances (not sure what the exact number is but I suspect its pretty big).

Bob has already backed off this issue, the only thing his test proves is that Crafty has not improved as much due to software. His test proves that and he is not claiming that his test proves anything else.

And when you think about it there is no reasonable test we could have done to settle the issue. Such a test really needed to be automated and using Crafty was the only reasonable test we could construct.

It's unfortunate that this does not address what the original debate was about, but I don't have any suggestions on how to properly do this.

As I said in the new thread I created, we should be able to get a pretty good answer to the original question by taking into account the distance from crafty to the best chess engine both in 1995 and 2010. For me the conclusion was that software contributed almost as much as hardware in terms of elo, but probably a bit less.

Actually, I believe the contribution is about the same all things considered. Bob's test actually put that in doubt for me, it makes it seem that software is the biggest contributor when you add the Rybka factor, parallel efficiency and so on. But there are so many variables to consider that in my mind have no clear answer that I think this is useless and why I have refrained from commenting. To me the test surely "demonstrates" that at least within a year or two (in either direction) you can say that software and hardware are equal contributors. If you think you will get Bob to agree that this is the case - don't hold your breath waiting.

If you'd only read, I believe I have already said _exactly_ that. +300 still leaves the hardware with a 140 point (or maybe more, I could not go back in time far enough to see if those 1995 doublings would be more than +80 or not. I suspect they would. I am not sure +300 is correct.

A 200 point rating advantage is still an advantage that says you will win about 3 of every 4 games played, so it is not insignificant. I'm right now working on re-calibrating all my tests using stockfish 1.8, which is real close to Rybka. That will at least give a reasonable estimate of the gap between Crafty and Rybka, and an exact measure of the gap between Crafty and Stockfish when no book or learning is allowed.

Fair enough.

I'm not that interested in getting an exact answer because it's a fools errand. Both software and hardware don't improve on a smooth curve but in big sudden jumps so what is the use trying to nail this down precisely? It will alway be subject to interpretation.

[bob]

What is that based on? We had dozens of bitboard programs prior to Rybka. I do not know of any after Rybka that were not there before it was modified to use bitboards... There is a limit to how much credit one can give to Rybka. It's certainly strong. It was not the "final coronation" for bitboard programs. It came to that party _way_ late.

I'm not sure if this is based on my comment that bitboards was a fad started by Rybka or not - but in either case I want to clarify the issue as I see it as the only thing worth commenting on in this thread - of which everything has been rehashed to death.

I agree with Bob that bitboard programs have been around a very long time. I was using them myself way before Rybka came out and Crafty always used them.

But that was not my point and I want that to be understood. Whether it was the right way to do it or not, what made it extremely popular was the fact that this new "rock star" Rybka was doing it. Personally I believe it's the best way to write a 64 bit program but that is not what made it so very popular.

Crafty also was doing it but until fairly recently almost all the top programs for 32 bit computers were 32 bit programs. That certainly did not sell anyone on the idea that 64 bit was the best way to write a program for a 32 bit machine.

Personally, I don't believe it is. I cannot prove that or back it up in any way but I would like to add that neither can you.

Except that I ran, quite successfully, on 32 bit hardware until 64 bit stuff came along. Won a couple of CCT online events using 32 bit hardware with bitboard Crafty. So it was definitely competitive. I even won one on my old sucky pentium 4 dual-cpu box, probably one of the worst things ever to come out of Intel.

Bitboards are not a losing proposition on 32 bit hardware. Again, we can compare move generation speed, evaluation speed, etc. Doesn't matter to me. I did the comparisons in 1994 and found that bitboards were faster. Slate, without even rotated bitboards, decided they were faster too, on non-64 bit hardware.

This "they are not efficient on 32 bit hardware" is an old urban legend created by who knows. But it is just legend, and not fact. Or else I was quite lucky in some of my 32 bit competitions.... Probably would have been bad on an older 486 box with no super-scalar and no cache to speak of. But once we got the super-scalar Pentium, the extra instructions became moot and kept the second pipe busier than a normal program could to hide the overhead. The overhead then disappeared completely when we got 64 bits, but with the loss of the extra instructions, the 64 bit gains were not always as high as expected since there were fewer instructions to execute in parallel.

[bob]

I guess I understand what you are saying, and I have always agreed with you (in fact I consider it quite obvious) that hardware advances have contributed more than software, but by using modern crafty instead of a top program you seem to be lopping off around 300 elo of today's advances (not sure what the exact number is but I suspect its pretty big).

Bob has already backed off this issue, the only thing his test proves is that Crafty has not improved as much due to software. His test proves that and he is not claiming that his test proves anything else.

And when you think about it there is no reasonable test we could have done to settle the issue. Such a test really needed to be automated and using Crafty was the only reasonable test we could construct.

It's unfortunate that this does not address what the original debate was about, but I don't have any suggestions on how to properly do this.

As I said in the new thread I created, we should be able to get a pretty good answer to the original question by taking into account the distance from crafty to the best chess engine both in 1995 and 2010. For me the conclusion was that software contributed almost as much as hardware in terms of elo, but probably a bit less.

Actually, I believe the contribution is about the same all things considered. Bob's test actually put that in doubt for me, it makes it seem that software is the biggest contributor when you add the Rybka factor, parallel efficiency and so on. But there are so many variables to consider that in my mind have no clear answer that I think this is useless and why I have refrained from commenting. To me the test surely "demonstrates" that at least within a year or two (in either direction) you can say that software and hardware are equal contributors. If you think you will get Bob to agree that this is the case - don't hold your breath waiting.

If you'd only read, I believe I have already said _exactly_ that. +300 still leaves the hardware with a 140 point (or maybe more, I could not go back in time far enough to see if those 1995 doublings would be more than +80 or not. I suspect they would. I am not sure +300 is correct.

A 200 point rating advantage is still an advantage that says you will win about 3 of every 4 games played, so it is not insignificant. I'm right now working on re-calibrating all my tests using stockfish 1.8, which is real close to Rybka. That will at least give a reasonable estimate of the gap between Crafty and Rybka, and an exact measure of the gap between Crafty and Stockfish when no book or learning is allowed.

Fair enough.

I'm not that interested in getting an exact answer because it's a fools errand. Both software and hardware don't improve on a smooth curve but in big sudden jumps so what is the use trying to nail this down precisely? It will alway be subject to interpretation.

I agree. And all I have concluded so far is that best case, hardware provided 2/3 of the advancement over the past 15 years. Worst case it would be pretty close to 50-50. Somewhere in there lies the truth. About the only thing one can conclude is that software did not provide more of the Elo than hardware. Which is at least a step in the right direction.