Computer Chess Progress: Stockfish 7 vs Ruffian 1.0.5

[fierz]

Dear all,

I have tried to add an experimental data point to the discussion on computer chess progress. Originally, I wanted to use Rebel Century or Rebel 6, but it turned out they didn't really adhere to Arena's time limit

So I ended up running Ruffian 1.05 (2002 if I am not mistaken, not the very best engine around then, but in the top group) against Stockfish 7 32-bit. I did this on an older Laptop (Pentium M, 1.5GHz, Win XP), 1000 games (starting from Ed Schröders work1.pgn) at 1s/move.

Ruffian was getting a bit more than 1MN/s, SF about half. Ruffian is 14 years old (so not quite the 20 years I had in mind, but close) and in the CCRL 40/40 list, stockfish 7 64 bit is on 3246, ruffian on 2608, a 638 point rating difference. SF 2.2.2 is on the list with both 32- and 64 bit versions, and has a 26 point rating difference there, so I could estimate a 612 point rating difference between SF7 and ruffian 1.0.5 according to the CCRL 40/40 list.

I would put 1s/move on the Pentium M 1.5GHz to around 100 times (s)lower than the CCRL 40/40 processing power. Theoretically it's possible that at lower NPS some things don't work any more. It's also possible that the rating list is sort of inconsistent because top engines are not playing older engines any more.

My experiment says otherwise: Stockfish won 974-26, Arena calculates a rating difference of 604 elo, really surprisingly close to the 612 of the CCRL list.

So I think I'll claim that this backs up my original claim of 750 elo over 20 years due to software alone, and that this would theoretically have been possible 20 years ago too - the algorithms just were not there. It would be nice to try even older top engines (Shredder or Fritz 5.32) but I have no access to them.

cheers
Martin

[yurikvelo]

Intel CPU history:

04.01.1996 - Pentium 166
10.06.1996 - Pentium 200
07.05.1997 - Pentium II 300
18.08.1997 - Pentium Pro 200
15.04.1998 - Pentium II 400
26.02.1999 - Pentium III 500
17.03.1999 - Xeon III 550
02.08.1999 - Pentium III 600
12.01.2000 - Pentium III 800
08.03.2000 - Pentium III 1000
20.11.2000 - Pentium 4 1500
23.04.2001 - Pentium 4 1700
02.06.2001 - Pentium 4 1800
27.08.2001 - Pentium 4 2000
07.01.2002 - Pentium 4 2200
02.04.2002 - Pentium 4 2400
26.08.2002 - Pentium 4 2600, FSB400
14.11.2002 - Pentium 4 3060, HT
12.03.2003 - Pentium M 1500 Centrino
23.09.2003 - Pentium 4 3200, FSB533

You run SF7 at pretty modern CPU (1M L2 cache, FSB400, SSE2)

Techniques which work well on 500-1000 kN/move might not work at 10-30 kn/move

P.S. I have alive PIII-500 (512 SDRAM, Windows XP) and could run some tests if there is interest

[yurikvelo]

I'll claim that this backs up my original claim of 750 elo over 20 years due to software alone

Hypo 1: At old CPU modern engine might slowdown much more than old engine.
If Ruffian is 2x faster than SF7 at modern Centrino CPU, it might be 5x faster at P-I

Hypo 2: Search techniqes which gain ELO at 100+ knodes per move might decrease ELO at 10+ knodes per move

[abulmo]

So I think I'll claim that this backs up my original claim of 750 elo over 20 years due to software alone, and that this would theoretically have been possible 20 years ago too - the algorithms just were not there. It would be nice to try even older top engines (Shredder or Fritz 5.32) but I have no access to them.

The strength of stockfish is not only the result of algorithms, but mostly the result of fine tuning and massive testing at hyperbullet time control, which was not possible to achieve with 20 year old hardware.

[Laskos]

massive testing at hyperbullet time control, which was not possible to achieve with 20 year old hardware.

Did they try hyper-bullet back then? I am not aware of any tool facilitating that. Even depth=1 testing can be of some use, and I am again not aware of ancient tries, it could have been done even in 1960. In fact my guess is that they were very sloppy testers, and this is their fault, CC is an empirical field.

[Joost Buijs]

It would be nice to try even older top engines (Shredder or Fritz 5.32) but I have no access to them.

I keep a lot of engines from that period in a backup, most of the stronger ones are CB-dll format and not UCI compliant, I guess the only way to use them is with a CB-GUI.

[yurikvelo]

So I ended up running Ruffian 1.05 (2002 if I am not mistaken, not the very best engine around then, but in the top group)

2002 Nov. Version 1.05 o It's basically the same as v1.0.1 with some bugfixes
2002-09-29 Version 1.0.1 o Fixed broken UCI interface. Ponder on should work now.
2002-09-22 Version 1.0.0 o First public release.


Offical results
---------------
2003, 5th CCT, Internet Champion 7/9 points, ranking 1 of 45
2003, 23rd Dutch Open, Champion 8.5/11 points, ranking 1 of 14


Did you disable supplied "Ruffian.bok" D=26 book?

[abulmo]

massive testing at hyperbullet time control, which was not possible to achieve with 20 year old hardware.

Did they try hyper-bullet back then? I am not aware of any tool facilitating that. Even depth=1 testing can be of some use, and I am again not aware of ancient tries, it could have been done even in 1960. In fact my guess is that they were very sloppy testers, and this is their fault, CC is an empirical field.

Yes, sloppiness may be in cause. I just randomly open a typical paper of this era:
"Each match consisted of 40 games, with the time controls
set to 40 moves in 5 minutes.[...] The preliminary results indicate that
this method has some potential and is definitely worth
refining and experimenting with, but more study is def-
initely needed to reach a final verdict." (1)
The "preliminary result" was 24 - 16, which of course means nothing.

However, the heart of the problem was the lack of inexpensive fast multicore hardware 20 years ago. To believe that a 15knodes capable engine would have been able to play 1500 nodes in 0.1 sec (average time per move in hyperbullet games) is a misconception. in 0.1s, a program might have just initialize its hashtable and its eval, with no time remained to do the search.

--
(1) Bjornsson, Y. and Marsland, T. A. (2000b). Selective
depth-first search methods. In van den Herik, H. J. and Iida, H., editors,
Games in AI Research, pages 31–46.

[Stan Arts]

For a long time there was rather the common belief that plenty of clever tricks were left to be discovered. And that clever secret programming and search tricks along with adding tons of specific knowledge would lead to the better programs.

You do not resort to endless testing if you do not have to. It sucks the creative fun right out of computerchess.

Still around 2005 when I joined some tournaments in Leiden all the talk was crazy creative ideas, not endless tuning.

After Fruit it became more apparant that maybe the strong guys did not have all these clever crazy tricks and brilliant evaluationfunctions. But I'm sure a lot of guys were already aware of that long before then and that the Shredders, Ruffians etc. were a result of that.

[jdart]

Yeah, I agree: I think it was trial and error back then and some sloppy testing. I used to run an overnight set of test suites. But that was a quite imperfect measure of strength. Still, with the hardware I had then, I could not run a useful series of matches in the same time period. Single core processors were the norm then.

--Jon