Computer chess progress over say the last 20 years?

[fierz]

Dear all,

I would like to produce a graph showing the progress of the top chess computer over time - in one version as absolute numbers (elo vs year), and in another version, pure algorithmic progress (elo vs year of release, on identical hardware). I can find a lot of information on the web - for example, the CCRL rating list has lots of current and older programs running on equal hardware to compare. What I'm lacking though is a list of which chess program was best in a given year (to search the top program per year in the CCRL list).

Has anyone got an idea where I could find this info, or maybe already compiled such a list of progress in computer chess? I would especially like to be able to compare the influence that better hardware had and the influence that better software had.

One example of what I would like to do: using the CCRL (40/4) list but maybe there is some better resource?

2016: best engine is Stockfish with a single-CPU rating of 3246
2005: Fruit 2.1 was one of the best engines around, with a rating of 2693
2003: Ruffian 1.05 (?) was one of the best engines , with a rating of 2608

I would like to extend this list with more years, and further back, but I'm unsure of which engines were top when - can anyone help?

I find it very interesting that pure algorithmic progress between 2003 and 2016 yielded 600 rating points! Guesstimating 60 Elo for a speed doubling, 18 Months as doubling time from Moore's law, and 13 years, I get ~500 elo for hardware improvement in that time, so software appears to have made more progress than hardware. I would also be interested to hear your thoughts on what makes the difference between Stockfish and e.g. Ruffian - what was invented algorithm-wise in the last 10 years (LMR? what else?) and how much did it contribute to the improvement? Or is it all due to better testing?

best regards
Martin

[jdart]

Not sure how to assemble the data you want - you really want something like snapshots of the rating lists by year. Maybe the Wayback Machine can give you that? (https://archive.org/web/).

For a long time commercial programs such as Shredder and Fritz were superior to any of the amateur or open-source programs. Up till 2005 or so at least the ICCF Championships (https://en.wikipedia.org/wiki/World_Com ... ampionship) give a pretty good idea of who was on top when. But in the past 10 years or so the amateur programs have caught up. Now the ICCF events are not a good indicator. Stockfish does not play in them. Also Houdini was on top for while but it never participated.

[bob]

Dear all,

I would like to produce a graph showing the progress of the top chess computer over time - in one version as absolute numbers (elo vs year), and in another version, pure algorithmic progress (elo vs year of release, on identical hardware). I can find a lot of information on the web - for example, the CCRL rating list has lots of current and older programs running on equal hardware to compare. What I'm lacking though is a list of which chess program was best in a given year (to search the top program per year in the CCRL list).

Has anyone got an idea where I could find this info, or maybe already compiled such a list of progress in computer chess? I would especially like to be able to compare the influence that better hardware had and the influence that better software had.

One example of what I would like to do: using the CCRL (40/4) list but maybe there is some better resource?

2016: best engine is Stockfish with a single-CPU rating of 3246
2005: Fruit 2.1 was one of the best engines around, with a rating of 2693
2003: Ruffian 1.05 (?) was one of the best engines , with a rating of 2608

I would like to extend this list with more years, and further back, but I'm unsure of which engines were top when - can anyone help?

I find it very interesting that pure algorithmic progress between 2003 and 2016 yielded 600 rating points! Guesstimating 60 Elo for a speed doubling, 18 Months as doubling time from Moore's law, and 13 years, I get ~500 elo for hardware improvement in that time, so software appears to have made more progress than hardware. I would also be interested to hear your thoughts on what makes the difference between Stockfish and e.g. Ruffian - what was invented algorithm-wise in the last 10 years (LMR? what else?) and how much did it contribute to the improvement? Or is it all due to better testing?

best regards
Martin

I don't think you can accurately make the claim that software made more progress. There are some things we do in software today solely because the hardware is so fast we don't get penalized as much as we would have 10 or 20 years ago. So clock doubling is but a part of the hardware equation. This phenomenon has been observed for 50 years now, in fact. Prior to chess 4.x, nobody could get away with exhaustive search, it was way too expensive and the 2-3 ply depths were not enough to beat even barely decent humans. Until the Cyber 176 came along. We've seen that happen several times now.

[Vinvin]

Dear all,

I would like to produce a graph showing the progress of the top chess computer over time - in one version as absolute numbers (elo vs year), and in another version, pure algorithmic progress (elo vs year of release, on identical hardware). I can find a lot of information on the web - for example, the CCRL rating list has lots of current and older programs running on equal hardware to compare. What I'm lacking though is a list of which chess program was best in a given year (to search the top program per year in the CCRL list).

Has anyone got an idea where I could find this info, or maybe already compiled such a list of progress in computer chess? I would especially like to be able to compare the influence that better hardware had and the influence that better software had.

One example of what I would like to do: using the CCRL (40/4) list but maybe there is some better resource?

2016: best engine is Stockfish with a single-CPU rating of 3246
2005: Fruit 2.1 was one of the best engines around, with a rating of 2693
2003: Ruffian 1.05 (?) was one of the best engines , with a rating of 2608

I would like to extend this list with more years, and further back, but I'm unsure of which engines were top when - can anyone help?

I find it very interesting that pure algorithmic progress between 2003 and 2016 yielded 600 rating points! Guesstimating 60 Elo for a speed doubling, 18 Months as doubling time from Moore's law, and 13 years, I get ~500 elo for hardware improvement in that time, so software appears to have made more progress than hardware. I would also be interested to hear your thoughts on what makes the difference between Stockfish and e.g. Ruffian - what was invented algorithm-wise in the last 10 years (LMR? what else?) and how much did it contribute to the improvement? Or is it all due to better testing?

best regards
Martin

Here are some data from 1995 to 2001 based on the SSDF rating list : http://www.talkchess.com/forum/viewtopi ... 801#532801

[mjlef]

Dear all,

I would like to produce a graph showing the progress of the top chess computer over time - in one version as absolute numbers (elo vs year), and in another version, pure algorithmic progress (elo vs year of release, on identical hardware). I can find a lot of information on the web - for example, the CCRL rating list has lots of current and older programs running on equal hardware to compare. What I'm lacking though is a list of which chess program was best in a given year (to search the top program per year in the CCRL list).

Has anyone got an idea where I could find this info, or maybe already compiled such a list of progress in computer chess? I would especially like to be able to compare the influence that better hardware had and the influence that better software had.

One example of what I would like to do: using the CCRL (40/4) list but maybe there is some better resource?

2016: best engine is Stockfish with a single-CPU rating of 3246
2005: Fruit 2.1 was one of the best engines around, with a rating of 2693
2003: Ruffian 1.05 (?) was one of the best engines , with a rating of 2608

I would like to extend this list with more years, and further back, but I'm unsure of which engines were top when - can anyone help?

I find it very interesting that pure algorithmic progress between 2003 and 2016 yielded 600 rating points! Guesstimating 60 Elo for a speed doubling, 18 Months as doubling time from Moore's law, and 13 years, I get ~500 elo for hardware improvement in that time, so software appears to have made more progress than hardware. I would also be interested to hear your thoughts on what makes the difference between Stockfish and e.g. Ruffian - what was invented algorithm-wise in the last 10 years (LMR? what else?) and how much did it contribute to the improvement? Or is it all due to better testing?

best regards
Martin

I don't think you can accurately make the claim that software made more progress. There are some things we do in software today solely because the hardware is so fast we don't get penalized as much as we would have 10 or 20 years ago. So clock doubling is but a part of the hardware equation. This phenomenon has been observed for 50 years now, in fact. Prior to chess 4.x, nobody could get away with exhaustive search, it was way too expensive and the 2-3 ply depths were not enough to beat even barely decent humans. Until the Cyber 176 came along. We've seen that happen several times now.

I would be very surprised if most of the gain over the last say 15 years was not due to better software. There is a ton of evidence for this, but if someone has a suitable "high end" machine from 15 years ago, I could compile Komodo for it and have it run against the best software of its day. It would be an interesting experiment. I predict the highly selective search schemes we use now will beat any of the old programs from 15 years ago. Granted there are differences in memory but PCs in 2000 had 128 MB or memory, running at say 600 MHz. Eventually if you go far enough in the past there will not be enough memory to do some of the things we do now, but I think a stripped down version of Komodo would do well. I suspect this since it runs well on a cheap Android phone without much memory or hig processor speed. Albert Silver even showed that the strongest program from years ago running on today;s hardware gets beat by Komodo running on a cell phone getting 1/50th the number of nodes per seconds as on the PC:

http://en.chessbase.com/post/komodo-8-t ... -challenge

Frankly, hardware speeds are not increasing much lately, like they did in the past. So I think most gains are due to better software. Anyone willing to run the experiment, let me know.

[Joost Buijs]

Dear all,

I would like to produce a graph showing the progress of the top chess computer over time - in one version as absolute numbers (elo vs year), and in another version, pure algorithmic progress (elo vs year of release, on identical hardware). I can find a lot of information on the web - for example, the CCRL rating list has lots of current and older programs running on equal hardware to compare. What I'm lacking though is a list of which chess program was best in a given year (to search the top program per year in the CCRL list).

Has anyone got an idea where I could find this info, or maybe already compiled such a list of progress in computer chess? I would especially like to be able to compare the influence that better hardware had and the influence that better software had.

One example of what I would like to do: using the CCRL (40/4) list but maybe there is some better resource?

2016: best engine is Stockfish with a single-CPU rating of 3246
2005: Fruit 2.1 was one of the best engines around, with a rating of 2693
2003: Ruffian 1.05 (?) was one of the best engines , with a rating of 2608

I would like to extend this list with more years, and further back, but I'm unsure of which engines were top when - can anyone help?

I find it very interesting that pure algorithmic progress between 2003 and 2016 yielded 600 rating points! Guesstimating 60 Elo for a speed doubling, 18 Months as doubling time from Moore's law, and 13 years, I get ~500 elo for hardware improvement in that time, so software appears to have made more progress than hardware. I would also be interested to hear your thoughts on what makes the difference between Stockfish and e.g. Ruffian - what was invented algorithm-wise in the last 10 years (LMR? what else?) and how much did it contribute to the improvement? Or is it all due to better testing?

best regards
Martin

I don't understand your statement.
When you subtract 2608 from 3246 there is a total increase of 638 Elo.
If the increase due to hardware alone is ~500 Elo the increase due to software is only ~138 Elo.
Shredder and Fritz were in 2003 clearly stronger than Ruffian, that makes it even look worse.
During the past 13 years the increase due to hardware is probably way less than 500 Elo, I have the feeling that it is something like 3 to 4 doublings (200-300 Elo).

It is virtually impossible to compare ratings form the past with current ratings.
In the past the main computer rating-list was the one from SSDF, which was more or less calibrated to human standards.
On the other hand the CCRL seems to exaggerate the ratings quite a bit.
It is like comparing watermelons to bananas.

[fierz]

Thanks for the suggestion with the waybeack machine, I will see what I can find and post it here!

[fierz]

CCRL tests all engines on the same hardware. So there is no difference between the hardware of Stockfish and Ruffian, the Elo difference is only due to software.

At the same time, hardware has been getting faster, and this gives an additional elo increase.

My feeling is the algorithms progressed faster than the hardware; and I also saw those results of modern programs on mobile phones beating old programs on current hardware, which makes me think that Bob's statement about "you couldn't do that back then" is untrue.

What would be really nice is a handicap match of Stockfish vs. some old program (like Fruit), where the thinking time of Stockfish is adjusted down until the programs are equally strong, simulating Stockfish on old hardware. I wonder how much you would have to slow it down!?

[Joost Buijs]

CCRL does not always test on the same hardware, the only thing they do is to adjust thinking time to an equivalent of an AMD64 using Crafty as a baseline which is something completely different.
The idea seems nice, but in practice this introduces a lot of error, not every engine will react the same to this.
The only way to get comparable ratings is to have ALL engines play each other on hardware that was common in the past and I'm sure that will never happen.

If there really was an increase of 500 Elo due to hardware then Ruffian would have had an Elo of ~2100 on the hardware from 2003.
The best programs of 2003 on hardware of 2003 were already quite comparable to the level of human GM's, so somewhere something goes wrong with all the assumptions.

[Ajedrecista]

Hello Martin:

Are you the programmer of Cake and CheckerBoard GUI? If so, thank you very much! I have used it many times.

I like your idea but I want to aware of something: it is familiar to me that CCRL (or was/were other rating list/s?) substracted Elo from their ratings once or twice (like -100 Elo each time). I found the following reference:

Chess Engine Rating Deflation at the CCRL Academy.

It is dated from 2013 but I think it is not the only deflation.

That is, if I were right (a big conditional) and there was a 100 Elo substraction in 2006 (just an example, it does not need to be true), then Fruit 2.1 had a rating of 2693 Elo in 2005, which would imply a rating of 2593 Elo nowadays.

Furthermore, make sure that ratings have been computed with the same rating programme under the same conditions (for example: using Bayeselo always and not changing from Bayeselo to Ordo).

I wish more people will write here in order to clarify or debunk my claim.

Regards from Spain.

Ajedrecista.