CRoberson wrote:While at the 2008 ACCA Pan American Computer Chess Championships,
Bob claimed he didn't believe software played a serious role in all the
rating improvements we've seen. He thought hardware deserved the
credit (assuming I understood the statement correctly. We were jumping
across several subjects and back that night.).
I beleive software has had much to do with it for several reasons.
I will start with one. The EBF with only MiniMax is 40. With Alpha-Beta
pruning, it drops to 6. In the early 1990's, the EBF was 4. Now, it is 2.
Dropping the EBF from 2 to 4 is huge. Lets look at a 20 ply search.
The speedup of EBF=2 vs EBF=4 is:
4^20/2^20 = 2^20 = 1,048,576
So, that is over a 1 million x speed up. Has hardware produced that much
since 1992?
Also, I believe eval improvements have caused an improvement in
rating scores.
An example of nonhardware improvements is on the SSDF rating list.
Rybka 1.0 beta score 2775 on a 450 MHz AMD.
Branching factor proves nothing because programs that do more pruning play weaker at fixed depth but I can say not based on branching factor that the improvement in software in the last years is very big and bigger than the improvement in hardware(not sure about improvement since 1992 because it is not clear how we define it but sure about improvement from 2005 to 2008).
Note that the tests of Bob can show only that hardware helped more than software for Crafty.
Tests of the SSDF showed the following results.
Rybka 3 A1200 - Deep Shredder 11 Q6600 20-19
Rybka 3 A1200 -Zappa Mexico II Q6600 20-20
A1200 = 1 x 1.2 GHz
Q6600 = 4 x 2.4 GHz
Note that both Zappa and Shredder are clearly stronger than Fruit that was the leading program in 2005 for single processor machines
I think that we can safely say that the software improvement in the last 3 years were more than 10:1 and I do not see hardware improvement of 10:1 in the last 3 years.
Uri
Can we stop with the amateurish comparisons/ Why pick different programs to compare. We had better and worse programs in 1995 as well. The question was, what have the software advances actually produced.
Feel free to name significant ones. Most would put null-move at the top, and LMR right behind it. Then we could factor in razoring, futility and extended futility from Heinz, but I already know those are very small improvements from testing by turning each off a month back or so.
what else is _significant_? Evaluation is not so interesting. I was doing passed pawn races in the 1970's. Outside passed pawns in 1995 in Crafty. So what _new_ thing since 1995 is such a big contributor? No hand-waving, no talking about ideas that _might_ have been developed. Actual documented techniques...
CRoberson wrote:While at the 2008 ACCA Pan American Computer Chess Championships,
Bob claimed he didn't believe software played a serious role in all the
rating improvements we've seen. He thought hardware deserved the
credit (assuming I understood the statement correctly. We were jumping
across several subjects and back that night.).
I beleive software has had much to do with it for several reasons.
I will start with one. The EBF with only MiniMax is 40. With Alpha-Beta
pruning, it drops to 6. In the early 1990's, the EBF was 4. Now, it is 2.
Dropping the EBF from 2 to 4 is huge. Lets look at a 20 ply search.
The speedup of EBF=2 vs EBF=4 is:
4^20/2^20 = 2^20 = 1,048,576
So, that is over a 1 million x speed up. Has hardware produced that much
since 1992?
Also, I believe eval improvements have caused an improvement in
rating scores.
An example of nonhardware improvements is on the SSDF rating list.
Rybka 1.0 beta score 2775 on a 450 MHz AMD.
Branching factor proves nothing because programs that do more pruning play weaker at fixed depth but I can say not based on branching factor that the improvement in software in the last years is very big and bigger than the improvement in hardware(not sure about improvement since 1992 because it is not clear how we define it but sure about improvement from 2005 to 2008).
Note that the tests of Bob can show only that hardware helped more than software for Crafty.
Tests of the SSDF showed the following results.
Rybka 3 A1200 - Deep Shredder 11 Q6600 20-19
Rybka 3 A1200 -Zappa Mexico II Q6600 20-20
A1200 = 1 x 1.2 GHz
Q6600 = 4 x 2.4 GHz
Note that both Zappa and Shredder are clearly stronger than Fruit that was the leading program in 2005 for single processor machines
I think that we can safely say that the software improvement in the last 3 years were more than 10:1 and I do not see hardware improvement of 10:1 in the last 3 years.
Uri
Can we stop with the amateurish comparisons? Why pick different programs to compare. We had better and worse programs in 1995 as well. The question was, what have the software advances actually produced.
Feel free to name significant ones. Most would put null-move at the top, and LMR right behind it. Then we could factor in razoring, futility and extended futility from Heinz, but I already know those are very small improvements from testing by turning each off a month back or so.
what else is _significant_? Evaluation is not so interesting. I was doing passed pawn races in the 1970's. Outside passed pawns in 1995 in Crafty. So what _new_ thing since 1995 is such a big contributor? No hand-waving, no talking about ideas that _might_ have been developed. Actual documented techniques...
CRoberson wrote:While at the 2008 ACCA Pan American Computer Chess Championships,
Bob claimed he didn't believe software played a serious role in all the
rating improvements we've seen. He thought hardware deserved the
credit (assuming I understood the statement correctly. We were jumping
across several subjects and back that night.).
I beleive software has had much to do with it for several reasons.
I will start with one. The EBF with only MiniMax is 40. With Alpha-Beta
pruning, it drops to 6. In the early 1990's, the EBF was 4. Now, it is 2.
Dropping the EBF from 2 to 4 is huge. Lets look at a 20 ply search.
The speedup of EBF=2 vs EBF=4 is:
4^20/2^20 = 2^20 = 1,048,576
So, that is over a 1 million x speed up. Has hardware produced that much
since 1992?
Also, I believe eval improvements have caused an improvement in
rating scores.
An example of nonhardware improvements is on the SSDF rating list.
Rybka 1.0 beta score 2775 on a 450 MHz AMD.
I believe my statement was more along the lines "Hardware has had a _larger_ influence over program performance increases over the past 20 years than the software has." This is based on my cluster testing where I now know just what some of the "great enhancements" have brought. If you'd like to pick one "revolutionary idea" (null-move? LMR? check extension? some evaluation concept? etc...) I can give it a test using crafty with and without, assuming crafty uses the idea you want to compare. I have not found a single +100 elo idea i Crafty. LMR is a modest improvement. I don't recall the exact amount at present but could compute it. Remember, just because we are physically searching deeper, today's "ply" is not comparable to "ply" of 20 years ago. Todays "plies" have significantly more errors in them due to various types of pruning and reductions going on...
I'll test whatever you think is the "biggest" to see what happens...
Your cluster tests can prove that hardware helped Crafty more than software.
They cannot prove that it is the case in general.
Uri
I'd claim Crafty is pretty representative of _most_ programs of today, null-move R=3, LMR, check extensions, iterative deepening, PVS, simple q-search + checks.. Etc...
Now if you want to claim most programs are far different from Crafty with respect to search space/techniques, feel free to do so. But I doubt that will convince anyone...
I think that the implementation of LMR may be different for different programs.
It may be interesting to do the same comparison for Glaurung to see how much rating Glaurung earns from LMR and null move.
CRoberson wrote:While at the 2008 ACCA Pan American Computer Chess Championships,
Bob claimed he didn't believe software played a serious role in all the
rating improvements we've seen. He thought hardware deserved the
credit (assuming I understood the statement correctly. We were jumping
across several subjects and back that night.).
I beleive software has had much to do with it for several reasons.
I will start with one. The EBF with only MiniMax is 40. With Alpha-Beta
pruning, it drops to 6. In the early 1990's, the EBF was 4. Now, it is 2.
Dropping the EBF from 2 to 4 is huge. Lets look at a 20 ply search.
The speedup of EBF=2 vs EBF=4 is:
4^20/2^20 = 2^20 = 1,048,576
So, that is over a 1 million x speed up. Has hardware produced that much
since 1992?
Also, I believe eval improvements have caused an improvement in
rating scores.
An example of nonhardware improvements is on the SSDF rating list.
Rybka 1.0 beta score 2775 on a 450 MHz AMD.
I believe my statement was more along the lines "Hardware has had a _larger_ influence over program performance increases over the past 20 years than the software has." This is based on my cluster testing where I now know just what some of the "great enhancements" have brought. If you'd like to pick one "revolutionary idea" (null-move? LMR? check extension? some evaluation concept? etc...) I can give it a test using crafty with and without, assuming crafty uses the idea you want to compare. I have not found a single +100 elo idea i Crafty. LMR is a modest improvement. I don't recall the exact amount at present but could compute it. Remember, just because we are physically searching deeper, today's "ply" is not comparable to "ply" of 20 years ago. Todays "plies" have significantly more errors in them due to various types of pruning and reductions going on...
I'll test whatever you think is the "biggest" to see what happens...
Your cluster tests can prove that hardware helped Crafty more than software.
They cannot prove that it is the case in general.
Uri
I'd claim Crafty is pretty representative of _most_ programs of today, null-move R=3, LMR, check extensions, iterative deepening, PVS, simple q-search + checks.. Etc...
Now if you want to claim most programs are far different from Crafty with respect to search space/techniques, feel free to do so. But I doubt that will convince anyone...
I think that the implementation of LMR may be different for different programs.
It may be interesting to do the same comparison for Glaurung to see how much rating Glaurung earns from LMR and null move.
Uri
if you have the time, go for it. I don't have the time to study the source to see what needs to be commented out. And I don't have the time to try this on every program available. Null-move doesn't give crafty any more or less improvement than any other program. LMR is not a giant step for mankind, so implementation details are going to be about small improvements or degradations, not big ones. I've never seen a quantitative analysis on precisely what null-move search does to a programs skill level, other than "it is clearly better". I just published the _precise_ data for this, played over enough games to be highly accurate.
bob wrote:Here are the results. Crafty-22.9X1 is normal crafty. 22.9X2 is normal except null-move completely commented out. 22.9X3 is normal except that LMR has been completely disabled. 22.9X4 is normal but with both LMR and null-move removed. the -1 or -2 just means run #1 or run#2 to give a fell for what kind of variation there is between runs.
normal is roughly 2637 in this test. Removing null-move or LMR drops this by approximately 40 Elo. Removing both drops the rating by around 120 Elo.
Null-move and LMR are the two biggest search enhancements of the past 15 years. And they added +120 Elo. I could always try normal crafty, but take the NPS from about 1M on this hardware (I only test with 1 cpu here) and back it down to about 75K which is what I was getting in 1996 on a pentium pro 200, roughly a factor of 15x and see how that impacts performance. Although I should probably factor in the single-core pentium pro vs a quad-core xeon for today, which runs around 10M nps, as that is a more representative example of what hardware speeds have done since 1996. So 75K to 10M is a factor of 24 or so. But something tells me that factor of 24-25x is _way_ more than 120 Elo...
You may be right but this is not a valid comparison! you should compare the improvement between Crafty model 1996 vs. Crafty model 2008, not the contribution of two single techniques as implemented in 2008.
What is the Elo difference between Crafty 1996 vs Crafty 2008 running in equal hardware? This is not the optimum either but it is closer to something more meaningful.
Miguel
Other suggestions???
BTW how many are surprised that removing both is only a 120 Elo loss???
bob wrote:Here are the results. Crafty-22.9X1 is normal crafty. 22.9X2 is normal except null-move completely commented out. 22.9X3 is normal except that LMR has been completely disabled. 22.9X4 is normal but with both LMR and null-move removed. the -1 or -2 just means run #1 or run#2 to give a fell for what kind of variation there is between runs.
normal is roughly 2637 in this test. Removing null-move or LMR drops this by approximately 40 Elo. Removing both drops the rating by around 120 Elo.
Null-move and LMR are the two biggest search enhancements of the past 15 years. And they added +120 Elo. I could always try normal crafty, but take the NPS from about 1M on this hardware (I only test with 1 cpu here) and back it down to about 75K which is what I was getting in 1996 on a pentium pro 200, roughly a factor of 15x and see how that impacts performance. Although I should probably factor in the single-core pentium pro vs a quad-core xeon for today, which runs around 10M nps, as that is a more representative example of what hardware speeds have done since 1996. So 75K to 10M is a factor of 24 or so. But something tells me that factor of 24-25x is _way_ more than 120 Elo...
You may be right but this is not a valid comparison! you should compare the improvement between Crafty model 1996 vs. Crafty model 2008, not the contribution of two single techniques as implemented in 2008.
No idea what you are talking about. This was an absolutely perfect attempt to quantify the effect of null-move and LMR. Null-move existed in 1995. Null-move existed in 1990. So that is _barely_ an enhancement from the last 20 years. Beal's original paper was somewhere in the 1988 range.
The discussion was not about 1988 vs 2008 for a 20 year span, it was about specific techniques developed during that time frame and how much they actually improve a program, vs the speed differential between machines over the last 20 years. I shortened it to 1995 because Crafty was available then, and I have a detailed record of what has been done over the last 13 years....
What is the Elo difference between Crafty 1996 vs Crafty 2008 running in equal hardware? This is not the optimum either but it is closer to something more meaningful.
]/quote]
I can probably produce that result. Let me see what the oldest version I have is, and see if I can figure out when it was current. I certainly know when the Jakarta version was done, since we know when the WCCC in Jakarta was played. But that would be an invalid test as well, because software "improvements" in the present context means "new ideas developed since XXX". Many of Crafty's improvements have been the result of tuning, but based on old ideas. I do not call those "software improvements" because they are not "general enhancements for all programs (such as null-move or LMR is) but are specific to a specific program's previously existing code...
Miguel
Other suggestions???
BTW how many are surprised that removing both is only a 120 Elo loss???
What eval changes have been made since 1992? At least the ones
suggested by Roman D. have been added.
I could name a few. "split passed pawns". -- now removed. Bishop pair scoring. -- now removed and presently based on an approach used in Cray Blitz dealing with mobility.
There's not a single evaluation term in Crafty I can think of that can not be found in the last version (1994) of Cray Blitz, and most can be found in the 1989 version that has been posted on the internet by Carey.
