SF Progression since Fishtest inception

[Adam Hair]

Here is a chart of Stockfish's month-by-month improvement since the birth of Fishtest, using SF 2.3.1 as the reference and including the official releases (and 2 or 3 versions that predate Fishtest).

CPUs: Xeon L5420
Time Control: 60" + 0.1"
Hash: 128 MB
Threads: 1
Openings: 50 positions 2 moves deep (a subset of the openings used in round 2 of TCEC)
Adjudication: resign @ 450 cp for 4 moves; draw after 120 moves if score is less than 50 cp for 4 moves
UI: cutechess-cli
Total number of games: 159600 (games available upon request)

Note: There is a distortion in the Elo ratings for Stockfish 3 and Stockfish 4 due to 7 time forfeits (all by Stockfish 3 when playing Stockfish 4) that have not been replayed yet.




I will be updating this chart each month at my my blog.

[Laskos]

Impressive test! I understand that there are 57 engines with 100 games against each other engine, total 5,600 games per engine, right? (57*56/2)*100 = 159,600
And the error margins for 95% confidence are 6-7 ELO points, right? As the monthly difference is on average about 10 ELO points, even successive results are statistically significant, without saying that those separated by several months are highly significant.

The pace of progress is impressive, probably only matched by Rybka 1 -> Rybka 3 progress. We also must note that Komodo's and Houdini's progress was also impressive, to almost match that of Stockfish. The last 5 or so years brought competent testing frameworks based on scientific method and a UI like Cutechess-Cli.

[Colin-G]

With your fast Xeon cpu, but short thinking time, (I assume 60" + 0.1" means a total of "1 minute plus 0.1 second per move" for the whole game) I was wondering what the average move depth was in the games.
Do you have that info?

[Laskos]

With your fast Xeon cpu, but short thinking time, (I assume 60" + 0.1" means a total of "1 minute plus 0.1 second per move" for the whole game) I was wondering what the average move depth was in the games.
Do you have that info?

The time control is not that short, it's the time control of Stockfish regression tests and LTC of Stockfish framework. Their STC time control is 15''+0.1''.

[Ozymandias]

The pace of progress is impressive, probably only matched by Rybka 1 -> Rybka 3 progress

I'd say that Rybka progress was quite different. Take a look at these snapshots from the CCRL 40/40, back in the day:

Code: Select all

1	Rybka 1.2 64-bit	2996	+23	−22	75.6%	−181.9	32.7%	739
 	Rybka 1.1 64-bit	2988	+24	−24	77.5%	−193.7	31.8%	663
 	Rybka 1.1 32-bit	2969	+25	−24	71.7%	−150.5	34.5%	577
 	Rybka 1.2 32-bit	2964	+33	−32	69.1%	−133.9	33.3%	330
 	Rybka 1.0 64-bit	2918	+22	−21	68.9%	−127.0	35.8%	745
 	Rybka 1.0 32-bit	2884	+19	−19	63.9%	− 93.4	38.0%	919
2	Shredder 10	     2878	+27	−26	61.6%	− 79.7	34.1%	469
3	Hiarcs X50 32-bit  2858	+29	−29	49.9%	  +2.2	41.1%	355
4	Fritz 9	         2849	+16	−16	55.5%	− 38.1	35.2%	1293
5	Fruit 2.2-2.2.1	 2843	+15	−15	55.4%	− 35.8	39.8%	1418
 	Toga II 1.1a	    2829	+18	−18	55.6%	− 36.2	37.4%	968

As soon as Rybka 1 was properly tuned (version 1.1), no one came between a hundred ELO points. Then we had some "slow" year, to bring SMP around and finally, another revolution with Rybka 3:

Code: Select all

1	Rybka 3 64-bit 4CPU	     3229	+24	−24	79.2%	−209.2	33.9%	684
 	Rybka 2.3.2a 64-bit 4CPU	3128	+15	−15	70.1%	−133.7	41.4%	1552
 	Rybka 2.2 64-bit 4CPU	   3111	+30	−29	73.3%	−157.6	38.6%	404
 	Rybka 2.1 64-bit 4CPU	   3082	+39	−37	72.9%	−161.5	36.5%	249
2	Naum 3.1 64-bit 4CPU	    3077	+20	−20	61.1%	− 71.9	43.7%	819
3	Zappa Mexico II 64b 4CPU	3074	+18	−18	61.4%	− 73.5	46.4%	947

What we have nowadays, is cumulative evolution. And that allowed SF to share 1st, somewhere along the road, first with Houdini 4, then with Komodo 8+ and currently, with the two of them. Nothing resembling the kind of dominance, Rybka showed.

[duncan]

The pace of progress is impressive, probably only matched by Rybka 1 -> Rybka 3 progress

I'd say that Rybka progress was quite different. Take a look at these snapshots from the CCRL 40/40, back in the day:

Code: Select all

1	Rybka 1.2 64-bit	2996	+23	−22	75.6%	−181.9	32.7%	739
 	Rybka 1.1 64-bit	2988	+24	−24	77.5%	−193.7	31.8%	663
 	Rybka 1.1 32-bit	2969	+25	−24	71.7%	−150.5	34.5%	577
 	Rybka 1.2 32-bit	2964	+33	−32	69.1%	−133.9	33.3%	330
 	Rybka 1.0 64-bit	2918	+22	−21	68.9%	−127.0	35.8%	745
 	Rybka 1.0 32-bit	2884	+19	−19	63.9%	− 93.4	38.0%	919
2	Shredder 10	     2878	+27	−26	61.6%	− 79.7	34.1%	469
3	Hiarcs X50 32-bit  2858	+29	−29	49.9%	  +2.2	41.1%	355
4	Fritz 9	         2849	+16	−16	55.5%	− 38.1	35.2%	1293
5	Fruit 2.2-2.2.1	 2843	+15	−15	55.4%	− 35.8	39.8%	1418
 	Toga II 1.1a	    2829	+18	−18	55.6%	− 36.2	37.4%	968

As soon as Rybka 1 was properly tuned (version 1.1), no one came between a hundred ELO points. Then we had some "slow" year, to bring SMP around and finally, another revolution with Rybka 3:

Code: Select all

1	Rybka 3 64-bit 4CPU	     3229	+24	−24	79.2%	−209.2	33.9%	684
 	Rybka 2.3.2a 64-bit 4CPU	3128	+15	−15	70.1%	−133.7	41.4%	1552
 	Rybka 2.2 64-bit 4CPU	   3111	+30	−29	73.3%	−157.6	38.6%	404
 	Rybka 2.1 64-bit 4CPU	   3082	+39	−37	72.9%	−161.5	36.5%	249
2	Naum 3.1 64-bit 4CPU	    3077	+20	−20	61.1%	− 71.9	43.7%	819
3	Zappa Mexico II 64b 4CPU	3074	+18	−18	61.4%	− 73.5	46.4%	947

What we have nowadays, is cumulative evolution. And that allowed SF to share 1st, somewhere along the road, first with Houdini 4, then with Komodo 8+ and currently, with the two of them. Nothing resembling the kind of dominance, Rybka showed.

what would hapen if stockfish was private like rybka, would there be more dominance ?

[Adam Hair]

Impressive test! I understand that there are 57 engines with 100 games against each other engine, total 5,600 games per engine, right? (57*56/2)*100 = 159,600
And the error margins for 95% confidence are 6-7 ELO points, right? As the monthly difference is on average about 10 ELO points, even successive results are statistically significant, without saying that those separated by several months are highly significant.

Thanks! You are right. 5600 games per engine, error ranges from 6 to 8 Elo.

Code: Select all

   # PLAYER                      :  RATING  ERROR  POINTS  PLAYED   (%)  CFS(%)     W     D     L  D(%)  OppAvg  OppErr  OppN  OppDiv
   1 Stockfish 221216 64         :       8      7  4190.0    5600  74.8      93  2947  2486   167  44.4    -212       6    56    56.0
   2 Stockfish 251116 64         :       1      7  4149.0    5600  74.1      55  2886  2526   188  45.1    -212       6    56    56.0
   3 Stockfish 8 64              :       0      7  4145.5    5600  74.0      57  2904  2483   213  44.3    -212       6    56    56.0
   4 Stockfish 220916 64         :      -1      7  4140.5    5600  73.9      87  2880  2521   199  45.0    -212       6    56    56.0
   5 Stockfish 221016 64         :      -7      7  4108.0    5600  73.4     100  2798  2620   182  46.8    -211       6    56    56.0
   6 Stockfish 190816 64         :     -35      7  3937.0    5600  70.3      82  2605  2664   331  47.6    -211       6    56    56.0
   7 Stockfish 210616 64         :     -40      7  3910.5    5600  69.8      95  2513  2795   292  49.9    -211       6    56    56.0
   8 Stockfish 160716 64         :     -47      6  3863.5    5600  69.0      98  2520  2687   393  48.0    -211       6    56    56.0
   9 Stockfish 200516 64         :     -57      6  3800.5    5600  67.9      99  2349  2903   348  51.8    -211       6    56    56.0
  10 Stockfish 180416 64         :     -69      6  3725.5    5600  66.5      96  2271  2909   420  51.9    -210       6    56    56.0
  11 Stockfish 150316 64         :     -77      6  3674.0    5600  65.6      84  2198  2952   450  52.7    -210       6    56    56.0
  12 Stockfish 200116 64         :     -82      6  3644.5    5600  65.1      52  2147  2995   458  53.5    -210       6    56    56.0
  13 Stockfish 210216 64         :     -82      6  3643.0    5600  65.1      90  2184  2918   498  52.1    -210       6    56    56.0
  14 Stockfish 221215 64         :     -88      6  3605.5    5600  64.4      77  2109  2993   498  53.4    -210       6    56    56.0
  15 Stockfish 7 64              :     -91      6  3584.0    5600  64.0      78  2081  3006   513  53.7    -210       6    56    56.0
  16 Stockfish 231015 64         :     -94      6  3561.0    5600  63.6      55  2078  2966   556  53.0    -210       6    56    56.0
  17 Stockfish 211115 64         :     -95      6  3557.5    5600  63.5     100  2069  2977   554  53.2    -210       6    56    56.0
  18 Stockfish 190915 64         :    -116      6  3414.5    5600  61.0      91  1919  2991   690  53.4    -210       6    56    56.0
  19 Stockfish 200815 64         :    -122      6  3374.0    5600  60.3      74  1889  2970   741  53.0    -209       6    56    56.0
  20 Stockfish 230715 64         :    -125      6  3354.5    5600  59.9      66  1878  2953   769  52.7    -209       6    56    56.0
  21 Stockfish 210515 64         :    -127      6  3342.0    5600  59.7      94  1835  3014   751  53.8    -209       6    56    56.0
  22 Stockfish 170615 64         :    -134      6  3294.0    5600  58.8      97  1799  2990   811  53.4    -209       6    56    56.0
  23 Stockfish 180415 64         :    -142      6  3238.5    5600  57.8      96  1715  3047   838  54.4    -209       6    56    56.0
  24 Stockfish 210315 64         :    -150      6  3186.5    5600  56.9     100  1720  2933   947  52.4    -209       6    56    56.0
  25 Stockfish 220215 64         :    -162      6  3100.0    5600  55.4      98  1566  3068   966  54.8    -209       6    56    56.0
  26 Stockfish 6 64              :    -171      6  3034.5    5600  54.2      93  1543  2983  1074  53.3    -209       6    56    56.0
  27 Stockfish 6 RC1 64          :    -178      6  2989.0    5600  53.4      93  1511  2956  1133  52.8    -208       6    56    56.0
  28 Stockfish 221214 64         :    -184      6  2943.5    5600  52.6      53  1406  3075  1119  54.9    -208       6    56    56.0
  29 Stockfish 211114 64         :    -185      6  2941.0    5600  52.5      98  1471  2940  1189  52.5    -208       6    56    56.0
  30 Stockfish 151014 64         :    -194      6  2878.5    5600  51.4     100  1405  2947  1248  52.6    -208       6    56    56.0
  31 Stockfish 100814 64         :    -210      6  2765.5    5600  49.4      83  1367  2797  1436  49.9    -208       6    56    56.0
  32 Stockfish 240614 64         :    -214      6  2736.5    5600  48.9      56  1303  2867  1430  51.2    -208       6    56    56.0
  33 Stockfish 210914 64         :    -215      6  2731.5    5600  48.8      58  1261  2941  1398  52.5    -208       6    56    56.0
  34 Stockfish 200714 64         :    -216      6  2725.0    5600  48.7      99  1298  2854  1448  51.0    -208       6    56    56.0
  35 Stockfish 190514 64         :    -226      6  2654.0    5600  47.4      92  1240  2828  1532  50.5    -208       6    56    56.0
  36 Stockfish 210414 64         :    -232      6  2611.5    5600  46.6      93  1252  2719  1629  48.6    -207       6    56    56.0
  37 Stockfish 5 64              :    -238      6  2567.5    5600  45.8      96  1172  2791  1637  49.8    -207       6    56    56.0
  38 Stockfish 220214 64         :    -246      6  2514.0    5600  44.9      73  1134  2760  1706  49.3    -207       6    56    56.0
  39 Stockfish 220314 64         :    -249      6  2495.0    5600  44.6     100  1133  2724  1743  48.6    -207       6    56    56.0
  40 Stockfish 190114 64         :    -263      6  2398.0    5600  42.8     100  1053  2690  1857  48.0    -207       6    56    56.0
  41 Stockfish 191213 64         :    -288      6  2231.5    5600  39.8      97   950  2563  2087  45.8    -206       6    56    56.0
  42 Stockfish 191113 64         :    -296      6  2174.0    5600  38.8      51   866  2616  2118  46.7    -206       6    56    56.0
  43 Stockfish DD 64             :    -296      6  2173.5    5600  38.8     100   872  2603  2125  46.5    -206       6    56    56.0
  44 Stockfish 231013 64         :    -318      6  2031.0    5600  36.3      96   795  2472  2333  44.1    -206       6    56    56.0
  45 Stockfish 190913 64         :    -326      6  1977.0    5600  35.3     100   766  2422  2412  43.3    -206       6    56    56.0
  46 Stockfish 4 64              :    -359      7  1774.5    5600  31.7      98   660  2229  2711  39.8    -205       6    56    56.0
  47 Stockfish 230813 64         :    -369      7  1712.5    5600  30.6     100   592  2241  2767  40.0    -205       6    56    56.0
  48 Stockfish 220713 64         :    -390      7  1589.5    5600  28.4      89   505  2169  2926  38.7    -205       6    56    56.0
  49 Stockfish 220613 64         :    -396      7  1555.5    5600  27.8      99   466  2179  2955  38.9    -205       6    56    56.0
  50 Stockfish 19-05-13 64bit    :    -408      7  1485.5    5600  26.5     100   435  2101  3064  37.5    -204       6    56    56.0
  51 Stockfish 3 64bit           :    -445      7  1287.5    5600  23.0      79   337  1901  3362  33.9    -204       6    56    56.0
  52 Stockfish 19-04-13 64bit    :    -450      7  1265.0    5600  22.6      83   309  1912  3379  34.1    -204       6    56    56.0
  53 Stockfish 19-03-13 64bit    :    -455      7  1239.5    5600  22.1      90   325  1829  3446  32.7    -203       6    56    56.0
  54 Stockfish 21-02-13 64bit    :    -462      7  1205.0    5600  21.5      51   274  1862  3464  33.3    -203       6    56    56.0
  55 Stockfish 2.3.1 64bit       :    -462      8  1204.0    5600  21.5     100   311  1786  3503  31.9    -203       6    56    56.0
  56 Stockfish  130127 64bit     :    -481      8  1114.0    5600  19.9      99   241  1746  3613  31.2    -203       6    56    56.0
  57 Stockfish  121228 64bit     :    -494      8  1052.0    5600  18.8     ---   208  1688  3704  30.1    -203       6    56    56.0

White advantage = 33.87 +/- 0.62
Draw rate (equal opponents) = 64.06 % +/- 0.17

The pace of progress is impressive, probably only matched by Rybka 1 -> Rybka 3 progress. We also must note that Komodo's and Houdini's progress was also impressive, to almost match that of Stockfish. The last 5 or so years brought competent testing frameworks based on scientific method and a UI like Cutechess-Cli.

When I first joined CCC, proper and effective testing methods were beginning to be acknowledged by the general population of authors and testers (having been informed by the scientists and mathematicians), but there were few tools available. Lack of tools is no longer a problem. Building a GM-level chess engine has changed from being an amalgam of art and computer science to being an engineering process.

[Adam Hair]

With your fast Xeon cpu, but short thinking time, (I assume 60" + 0.1" means a total of "1 minute plus 0.1 second per move" for the whole game) I was wondering what the average move depth was in the games.
Do you have that info?

The CPUs are 2.5 GHz and 8 years old, so they are not fast relative to today's processors.

Here is the output from ProTools 1.6a:

Code: Select all

Engine                 Depth       Time   Games     Moves  Average Forfeit Book
Stockfish 19-04-13 6   16.54   94:44:11    5600    337030    1.01     0   11155 (1.99)
Stockfish  121228 64   16.22   92:50:34    5600    330401    1.01     0   11136 (1.99)
Stockfish  130127 64   16.42   92:30:28    5600    330072    1.01     0   11154 (1.99)
Stockfish 19-03-13 6   16.60   94:15:40    5600    336661    1.01     0   11137 (1.99)
Stockfish 3 64bit      16.68   94:52:10    5600    340789    1.00     7   11146 (1.99)
Stockfish 2.3.1 64bi   16.23   92:15:38    5600    334218    0.99     0   11129 (1.99)
Stockfish 21-02-13 6   16.50   92:44:04    5600    336634    0.99     0   11150 (1.99)
Stockfish 220713 64    17.05   94:49:04    5600    347803    0.98     0   11136 (1.99)
Stockfish 19-05-13 6   17.06   95:07:03    5600    349034    0.98     0   11147 (1.99)
Stockfish 230813 64    17.18   94:04:48    5600    345728    0.98     0   11145 (1.99)
Stockfish 220613 64    16.98   95:02:56    5600    350417    0.98     0   11121 (1.99)
Stockfish 4 64         17.22   94:45:36    5600    352257    0.97     0   11125 (1.99)
Stockfish 231013 64    18.28   92:25:04    5600    353525    0.94     0   11132 (1.99)
Stockfish 191113 64    18.60   93:51:07    5600    359855    0.94     0   11136 (1.99)
Stockfish 191213 64    18.75   94:01:55    5600    362028    0.94     0   11139 (1.99)
Stockfish DD 64        18.65   93:51:07    5600    361599    0.93     0   11149 (1.99)
Stockfish 8 64         20.06   86:24:01    5600    340892    0.91     0   12119 (2.16)
Stockfish 190816 64    19.20   87:57:14    5600    348267    0.91     0   12157 (2.17)
Stockfish 220916 64    19.83   86:54:56    5600    344475    0.91     0   12044 (2.15)
Stockfish 210216 64    19.47   90:31:10    5600    358809    0.91     0   12356 (2.21)
Stockfish 221016 64    20.05   86:29:11    5600    343150    0.91     0   12112 (2.16)
Stockfish 251116 64    20.09   85:52:35    5600    340911    0.91     0   12045 (2.15)
Stockfish 160716 64    18.88   87:59:26    5600    349472    0.91     0   12226 (2.18)
Stockfish 190913 64    17.75   89:29:49    5600    355809    0.91     0   11132 (1.99)
Stockfish 220314 64    19.03   94:02:17    5600    374161    0.90     0   12592 (2.25)
Stockfish 221216 64    20.19   86:15:14    5600    343898    0.90     0   12082 (2.16)
Stockfish 6 64         18.73   93:55:30    5600    375026    0.90     0   12468 (2.23)
Stockfish 6 RC1 64     18.62   94:06:45    5600    375875    0.90     0   12471 (2.23)
Stockfish 210616 64    19.57   87:12:27    5600    349617    0.90     0   12329 (2.20)
Stockfish 220214 64    19.22   93:02:47    5600    373030    0.90     0   12598 (2.25)
Stockfish 220215 64    18.78   94:02:08    5600    377274    0.90     0   12485 (2.23)
Stockfish 211114 64    18.47   94:02:51    5600    377480    0.90     0   12418 (2.22)
Stockfish 190114 64    18.98   92:35:54    5600    372178    0.90     0   12679 (2.26)
Stockfish 211115 64    18.68   90:56:13    5600    365589    0.90     0   12345 (2.20)
Stockfish 5 64         19.20   93:52:38    5600    379443    0.89     0   12685 (2.27)
Stockfish 200815 64    18.77   91:29:57    5600    369936    0.89     0   12482 (2.23)
Stockfish 100814 64    18.91   93:38:00    5600    378729    0.89     0   12624 (2.25)
Stockfish 230715 64    18.62   91:41:21    5600    371189    0.89     0   12325 (2.20)
Stockfish 210914 64    18.87   93:09:52    5600    377454    0.89     0   12566 (2.24)
Stockfish 221214 64    18.71   93:53:56    5600    380567    0.89     0   12482 (2.23)
Stockfish 200516 64    19.70   87:33:39    5600    354895    0.89     0   12324 (2.20)
Stockfish 240614 64    18.94   93:21:23    5600    378669    0.89     0   12577 (2.25)
Stockfish 190514 64    19.29   93:22:49    5600    378953    0.89     0   12697 (2.27)
Stockfish 200714 64    19.01   93:34:09    5600    379740    0.89     0   12659 (2.26)
Stockfish 210414 64    19.11   94:01:24    5600    381724    0.89     0   12708 (2.27)
Stockfish 210315 64    18.46   92:08:08    5600    375238    0.88     0   12461 (2.23)
Stockfish 180416 64    19.30   87:52:07    5600    358107    0.88     0   12300 (2.20)
Stockfish 180415 64    19.02   92:43:10    5600    378868    0.88     0   12567 (2.24)
Stockfish 231015 64    18.75   91:05:32    5600    372386    0.88     0   12290 (2.19)
Stockfish 170615 64    19.08   92:26:18    5600    378020    0.88     0   12478 (2.23)
Stockfish 190915 64    18.88   91:36:39    5600    375343    0.88     0   12340 (2.20)
Stockfish 200116 64    19.48   87:37:11    5600    359091    0.88     0   12294 (2.20)
Stockfish 210515 64    19.02   92:18:16    5600    379080    0.88     0   12431 (2.22)
Stockfish 151014 64    19.67   93:53:14    5600    386856    0.87     0   12499 (2.23)
Stockfish 221215 64    18.88   88:21:58    5600    364631    0.87     0   12322 (2.20)
Stockfish 150316 64    19.28   88:13:47    5600    364921    0.87     0   12317 (2.20)
Stockfish 7 64         18.89   88:27:35    5600    365877    0.87     0   12334 (2.20)


Time control comparison between engines

Depth     : Average search depth
Time      : Total time engine used
Moves     : Total moves engine played
Average   : Average time per move in centi-seconds
Forfeit   : Games engine lost due to time forfeit

List is sorted on Average Time indicating the engine that uses the most time tops.

[Ozymandias]

what would hapen if stockfish was private like rybka, would there be more dominance ?

As long as they were able to keep so many people involved, and no one leaked the sources... But those are two big conditionals.

[Adam Hair]

Updated list (up to April 22):

Code: Select all

   # PLAYER                      : RATING  ERROR   POINTS  PLAYED    (%)
   1 Stockfish 180317 64         :   19.3    8.9   4459.5    6000   74.3%
   2 Stockfish 200417 64         :   11.6    9.0   4412.5    6000   73.5%
   3 Stockfish 210117 64         :   11.2    9.1   4410.5    6000   73.5%
   4 Stockfish 190217 64         :    8.7    9.0   4395.0    6000   73.2%
   5 Stockfish 221216 64         :    7.3    9.2   4386.5    6000   73.1%
   6 Stockfish 8 64              :    0.0   ----   4341.0    6000   72.3%
   7 Stockfish 251116 64         :    0.0    9.1   4341.0    6000   72.3%
   8 Stockfish 220916 64         :   -1.8    8.8   4330.0    6000   72.2%
   9 Stockfish 221016 64         :   -5.9    8.8   4304.0    6000   71.7%
  10 Stockfish 190816 64         :  -36.2    8.7   4107.0    6000   68.5%
  11 Stockfish 210616 64         :  -41.3    9.2   4073.0    6000   67.9%
  12 Stockfish 160716 64         :  -47.8    8.9   4029.5    6000   67.2%
  13 Stockfish 200516 64         :  -58.9    8.5   3954.0    6000   65.9%
  14 Stockfish 180416 64         :  -69.9    8.7   3877.5    6000   64.6%
  15 Stockfish 150316 64         :  -76.5    8.7   3831.5    6000   63.9%
  16 Stockfish 210216 64         :  -80.9    8.7   3800.5    6000   63.3%
  17 Stockfish 200116 64         :  -82.3    8.7   3790.5    6000   63.2%
  18 Stockfish 221215 64         :  -87.4    8.5   3755.0    6000   62.6%
  19 Stockfish 7 64              :  -90.0    8.7   3736.0    6000   62.3%
  20 Stockfish 211115 64         :  -94.2    8.6   3706.0    6000   61.8%
  21 Stockfish 231015 64         :  -94.7    8.7   3703.0    6000   61.7%
  22 Stockfish 190915 64         : -115.8    8.5   3551.0    6000   59.2%
  23 Stockfish 200815 64         : -122.4    8.6   3503.0    6000   58.4%
  24 Stockfish 230715 64         : -125.2    8.3   3482.0    6000   58.0%
  25 Stockfish 210515 64         : -126.9    8.4   3469.5    6000   57.8%
  26 Stockfish 170615 64         : -134.2    8.5   3416.0    6000   56.9%
  27 Stockfish 180415 64         : -142.4    8.8   3356.0    6000   55.9%
  28 Stockfish 210315 64         : -151.7    8.5   3287.5    6000   54.8%
  29 Stockfish 220215 64         : -163.4    8.7   3201.0    6000   53.4%
  30 Stockfish 6 64              : -171.5    8.5   3141.0    6000   52.4%
  31 Stockfish 6 RC1 64          : -178.7    8.6   3087.5    6000   51.5%
  32 Stockfish 221214 64         : -183.8    8.6   3049.5    6000   50.8%
  33 Stockfish 211114 64         : -184.6    8.4   3044.0    6000   50.7%
  34 Stockfish 151014 64         : -193.7    8.7   2976.5    6000   49.6%
  35 Stockfish 100814 64         : -208.9    8.7   2864.0    6000   47.7%
  36 Stockfish 240614 64         : -215.2    8.9   2817.5    6000   47.0%
  37 Stockfish 210914 64         : -215.3    8.8   2817.0    6000   47.0%
  38 Stockfish 200714 64         : -217.1    8.6   2803.5    6000   46.7%
  39 Stockfish 190514 64         : -226.9    8.4   2731.5    6000   45.5%
  40 Stockfish 210414 64         : -232.8    8.4   2688.0    6000   44.8%
  41 Stockfish 5 64              : -238.9    8.6   2644.0    6000   44.1%
  42 Stockfish 220214 64         : -248.6    8.7   2573.5    6000   42.9%
  43 Stockfish 220314 64         : -250.6    8.4   2559.0    6000   42.6%
  44 Stockfish 190114 64         : -263.4    9.1   2467.0    6000   41.1%
  45 Stockfish 191213 64         : -287.7    8.8   2295.0    6000   38.2%
  46 Stockfish DD 64             : -295.7    8.8   2239.0    6000   37.3%
  47 Stockfish 191113 64         : -297.3    8.7   2228.0    6000   37.1%
  48 Stockfish 231013 64         : -318.3    9.0   2084.5    6000   34.7%
  49 Stockfish 190913 64         : -327.1    9.2   2026.0    6000   33.8%
  50 Stockfish 4 64              : -360.0    9.1   1812.5    6000   30.2%
  51 Stockfish 230813 64         : -369.0    9.1   1756.0    6000   29.3%
  52 Stockfish 220713 64         : -389.8    9.2   1629.5    6000   27.2%
  53 Stockfish 220613 64         : -395.8    9.1   1593.5    6000   26.6%
  54 Stockfish 19-05-13 64bit    : -409.5    9.3   1514.0    6000   25.2%
  55 Stockfish 3 64bit           : -445.8    9.7   1314.5    6000   21.9%
  56 Stockfish 19-04-13 64bit    : -449.5    9.7   1295.0    6000   21.6%
  57 Stockfish 19-03-13 64bit    : -455.4    9.5   1264.5    6000   21.1%
  58 Stockfish 21-02-13 64bit    : -461.7    9.9   1232.0    6000   20.5%
  59 Stockfish 2.3.1 64bit       : -461.8    9.6   1231.5    6000   20.5%
  60 Stockfish  130127 64bit     : -481.5   10.0   1134.5    6000   18.9%
  61 Stockfish  121228 64bit     : -493.7   10.2   1077.0    6000   17.9%