CCRL Blitz: 47 Free Single-CPU engines ranked

[Kirill Kryukov]

Hi all! 47 free single-CPU engines are now ranked in CCRL Blitz:

Code: Select all

 Rank                 Engine                  ELO   +    -   Score  AvOp  Games
    1 Rybka 1.0 Beta 64-bit                  2944  +17  -17  74.4% -178.2  1364
    2 Fruit 2.3 5-men-egbb                   2873  +21  -21  67.1% -121.7   832
    3 Spike 1.2 Turin                        2826   +7   -7  45.2%  +33.3  8993
    4 Naum 2.0 32-bit                        2796   +8   -8  48.8%   +3.2  5766
    5 Alaric 707                             2769  +14  -14  52.3%  -18.8  1951
    6 Glaurung 1.2.1 32-bit                  2758   +8   -8  42.0%  +56.6  5789
    7 Scorpio 1.91 (5-men-egbb)              2732  +16  -16  50.2%   -2.7  1294
    8 Movei 0.08.438(10 10 10)               2731  +18  -18  51.6%  -11.8  1031
    9 Pro Deo 1.2                            2721  +12  -12  43.3%  +50.8  2627
   10 Delfi 5.1                              2715  +12  -11  51.6%  -11.5  2766
   11 Slow Chess Blitz WV2.1                 2713   +9   -9  42.1%  +57.8  5154
   12 Zappa 1.1 64-bit                       2712  +16  -16  47.1%  +20.1  1376
   13 Frenzee 3.0 64-bit                     2710  +15  -16  49.4%   +4.2  1476
   14 Pharaon 3.5.1                          2700  +11  -11  39.2%  +80.1  3151
   15 WildCat 7                              2698  +11  -11  48.6%  +10.4  3261
   16 List 5.12                              2696  +11  -11  49.4%   +4.7  2927
   17 Colossus 2007c                         2688  +16  -16  47.6%  +16.9  1344
   18 SOS 5.1                                2682  +12  -12  46.8%  +23.0  2505
   19 Pseudo 0.7c                            2676  +11  -11  50.7%   -8.3  2997
   20 Ruffian 1.0.5                          2674  +13  -13  49.5%   +3.6  2286
   21 Aristarch 4.50                         2668  +12  -12  46.0%  +30.8  2707
   22 Booot 4.13.1                           2667  +14  -14  51.1%   -7.1  1682
   23 Petir 4.39                             2663  +13  -13  46.4%  +27.0  2160
   24 Jonny 2.83 32-bit                      2651  +14  -14  46.2%  +29.6  1980
   25 The Baron 1.8.1                        2649  +16  -16  48.8%   +9.0  1462
   26 BugChess2 1.5.2                        2646  +17  -17  52.3%  -17.3  1152
   27 Crafty 21.5 PS 64-bit                  2644  +15  -15  45.5%  +32.9  1615
28-29 Anaconda 2.0.1                         2641  +13  -14  51.1%   -7.8  1934
28-29 Fritz 5.32                             2641  +15  -15  52.5%  -20.2  1560
   30 Smarthink 0.17a                        2635  +15  -15  49.1%   +7.3  1435
   31 Ktulu 4.2                              2631  +17  -17  50.9%   -5.9  1149
   32 Thinker 4.7a                           2630  +12  -12  49.2%   +1.7  2312
33-34 AnMon 5.60                             2617  +14  -14  48.3%  +12.7  1796
33-34 Trace 1.37a                            2617  +15  -15  46.8%  +24.4  1544
   35 Ufim 8.02                              2615  +12  -12  50.9%   -9.2  2460
   36 Little Goliath Evolution 3.12 (TB On)  2613  +18  -18  48.6%   +9.1  1055
   37 E.T. Chess 18.11.05                    2608  +13  -13  54.4%  -38.0  2250
   38 Yace 0.99.87                           2607  +18  -18  48.2%  +11.0  1024
   39 Hamsters 0.5                           2583  +19  -19  47.0%  +19.0   948
   40 Green Light Chess 3.00                 2572  +20  -20  45.3%  +30.7   863
   41 Tao 5.6                                2564  +20  -20  44.4%  +39.1   856
   42 Amyan 1.597                            2562  +14  -14  46.2%  +25.9  1910
   43 Abrok 5.0                              2557  +19  -19  47.4%  +11.8   956
   44 Arasan 10.0 64-bit                     2551  +22  -22  42.6%  +50.6   733
   45 Kiwi 0.6d                              2513  +26  -26  54.3%  -32.3   523
46-47 Comet B68                              2478  +21  -21  36.4%  +96.5   821
46-47 Sjeng 12.13                            2478  +24  -24  33.9% +114.5   640

(My selection consists of well-tested engines only).

Link to rating list and crosstables

News:

1. Comet B68 and Sjeng 12.13 testing is complete. At the moment they are sharing #46-47, though they will probably drop out of 50 when more engines are tested.

2. BugChess2 1.5.2 testing is in process. Currently ranked #26 after 1152 games played. Great results so far!

3. I started to test Kiwi 0.6d. Still not much to say, current place is #45.

I'll remind that in this comparison each engine must play at least 30-games match with every other participant which is 16 ranks away from it (or closer). As a result, finished matches form a "diagonal" in the crosstable.

As always thanks to all testers helping to test free engines! And bigger thanks to the authors giving away such great programs for free!

Best,
Kirill

[Alessandro Scotti]

Hi Kirill,
thanks for the update! I downloaded yesterday the games database but I'm more than happy to redownload to get 5000 more games!

[Kirill Kryukov]

Hi Alessandro! Cheers!

What do you do with all those games, feed to Hamsters?

BTW, I was sad to see that Hamsters 0.5 lost a few points since last update, but such is life.

One thing I was always curious about is how close relatives are Hamsters and Kiwi. Soon we'll know.

Best,
Kirill

[Alessandro Scotti]

Hi Kirill,
I'm using the games in experiments that aim at improving Hamsters evaluation. No much luck so far, but still a lot of tests to do.
I did notice that Hamsters is not doing well in CCRL blitz, I would like to discover why but currently the previous experiments have priority because I would really like to have something completely "new" in Hamsters.
Speaking of which, there should be very little similarity with Kiwi! Both search and evaluation are in fact completely different and Hamsters is much more "tactical" than Kiwi... but I'm also curious to know if this is supported by hard data!

[Tony Thomas]

Note: if you have an UCI interface you might try this settings in stead of
defaults: PreExt=18, CheckExt=24, MateExt=18, OneRep=18, RecapType=Normal,
RecapExt=18, NullType=23. Not sure it's better, though. But it seems a good
alternative in shorter time controls. Let me know what you think is better
at long tc's (bas.hamstra@wxs.nl) or if you have other suggestions.

The above should have helped Tao a little to crawl in to top 40.

[Kirill Kryukov]

Hi Kirill,
I'm using the games in experiments that aim at improving Hamsters evaluation. No much luck so far, but still a lot of tests to do.
I did notice that Hamsters is not doing well in CCRL blitz, I would like to discover why but currently the previous experiments have priority because I would really like to have something completely "new" in Hamsters.

I see, glad to hear that the games are helpful! Looking forward to improved Hamsters!

Speaking of which, there should be very little similarity with Kiwi! Both search and evaluation are in fact completely different and Hamsters is much more "tactical" than Kiwi... but I'm also curious to know if this is supported by hard data!

It should be in the next update. I'm curious to see it too! (I'm now doing complete test for Kiwi 0.6d which means it should play with all engines 16 positions away or closer to it in my selection)

I did notice that Hamsters is not doing well in CCRL blitz

Not well compared to your own test results, to some else's test results, or to your hope?

[Kirill Kryukov]

Note: if you have an UCI interface you might try this settings in stead of
defaults: PreExt=18, CheckExt=24, MateExt=18, OneRep=18, RecapType=Normal,
RecapExt=18, NullType=23. Not sure it's better, though. But it seems a good
alternative in shorter time controls. Let me know what you think is better
at long tc's (bas.hamstra@wxs.nl) or if you have other suggestions.

The above should have helped Tao a little to crawl in to top 40.

The list of engines I am going to test has grown pretty long by now, so I am not sure when I will have time to re-test an already tested engine with different setting. I don't mind if anyone else did it though.

[Alessandro Scotti]

I did notice that Hamsters is not doing well in CCRL blitz

Not well compared to your own test results, to some else's test results, or to your hope?

Eh eh... I only hope the best for Hamsters!

However, in this case I am referring to CCRL 40/40. Here is the rating difference (with respect to Hamsters 0.5) of a few engines I test with:
SmarThink, blitz=+52, long=+20
Colossus2007c, blitz=+105, long=+47
Logic099x, blitz=+3, long=-24
Pseudo 0.7c, blitz=+93, long=+39
Petir 4.39, blitz=+80, long=+29
(I also test against Boot 4.13.1 and Trace 1.37 but can't find the data).

It seems there is a 40/50 elo difference when comparing the two performances. I test at 1+1 and my results are a lot closer to 40/40 than 40/4, that's why I think Hamsters might have a problem in handling that time control.

[Tony Thomas]

Note: if you have an UCI interface you might try this settings in stead of
defaults: PreExt=18, CheckExt=24, MateExt=18, OneRep=18, RecapType=Normal,
RecapExt=18, NullType=23. Not sure it's better, though. But it seems a good
alternative in shorter time controls. Let me know what you think is better
at long tc's (bas.hamstra@wxs.nl) or if you have other suggestions.

The above should have helped Tao a little to crawl in to top 40.

The list of engines I am going to test has grown pretty long by now, so I am not sure when I will have time to re-test an already tested engine with different setting. I don't mind if anyone else did it though.

If I had known that you were going to test it, I would have told you about the above setting. I copied it from the readme file, which almost no one reads. Looks like Tao is a discontinued project, so I wont waste time on it either. I would love to see some Crafty 21.7 64bit results when it comes out though.

[Jouni]

Great list, but the strongest Fruit = 05/11/03 is missing.

Jouni