Excellent scaling with time of Shredder 13 4CPU on CCRL

[Laskos]

It is easy to miss (I missed) comparing CCRL 40/4' with CCRL 40/40' lists that Deep Shredder 13 4CPU scales very well with time control. It is the only engine which scales abnormally outside error margins. I took top 16 engines on 4CPU from CCRL (Komodo 10.4 was used, too few data for other recent Komodo), and computed the following:

Code: Select all

 #  Name       40/4 40/40   Expected    Scaling
-------------------------------------------------
 1 Stockfish   3485  3389    3400.17    -11.17
 2 Houdini     3471  3385    3388.50     -3.50
 3 Komodo      3469  3384    3386.83     -2.83
 4 Fire        3328  3273    3269.33     +3.67
 5 Fizbo       3315  3258    3258.50      -.50
 6 Shredder    3288  3292    3236.00    +56.00
 7 Andscacs    3271  3241    3221.83    +19.17
 8 Gull        3260  3193    3212.67    -19.67
 9 Booot       3249  3223    3203.50    +19.50
10 Equinox     3246  3185    3201.00    -16.00
11 Chiron      3237  3205    3193.50    +11.50
12 Critter     3231  3169    3188.50    -19.50
13 Hannibal    3225  3168    3183.50    -15.50
14 Fritz       3220  3170    3179.33     -9.33
15 Nirvana     3207  3161    3168.50     -7.50
16 Rybka       3203  3155    3165.17    -10.17
-------------------------------------------------    
       Mean:   3294  3241
       Range:   282   234 

Interpolating formula between ratings:
CCRL 40/40' = 3241 + (CCRL 40/4' - 3294) / 1.20

There is an offset in the mean and the range, rating compression here to longer TC is about 1.20. For these 16 top 4CPU engines the interpolating formula is:
CCRL 40/40' = 3241 + (CCRL 40/4' - 3294) / 1.20

With that I computed the expected ratings at 40/40' from 40/4' ratings, and then comparing to CCRL 40/40' list, got the scaling. The 2SD error in scaling is about 20-25 ELO points, due to statistical errors in CCRL lists and interpolation. The only engine scaling outside error margins is Deep Shredder 13 4CPU, gaining a whopping 56 ELO points to 10x longer time control compared to other engines. Probably Andscacs and Boot scale pretty well (+19), but are within 2SD error margins.

Just wanted to share the observation. Might be interesting to see Deep Shredder in TCEC.

[Thomas Lagershausen]

After looking at the cegt 40/120 ratinglist i see no danger for Stockfish at TCEC 10 by Shredder.

http://www.cegt.net/40120new/40_120_rat ... liste.html

The gap to the regular Stockfish 8 is 159 elopoints at tournamenttimecontrol.

Furthermore the softwaredevelopment by "Team Stockfish" is faster than the onemanshow of Shredder.

So no competition from this side for Stockfish.

[Laskos]

After looking at the cegt 40/120 ratinglist i see no danger for Stockfish at TCEC 10 by Shredder.

http://www.cegt.net/40120new/40_120_rat ... liste.html

The gap to the regular Stockfish 8 is 159 elopoints at tournamenttimecontrol.

Furthermore the softwaredevelopment by "Team Stockfish" is faster than the onemanshow of Shredder.

So no competition from this side for Stockfish.

I was talking of 4CPU performance. Also, CCRL 40/40 on 4 cores is pretty equivalent to 40/120 on one core of CEGT (if both are adjusting to the old AMD speed), and the error margins in CEGT list are almost twice as large. I am not saying that Shredder will beat Stockfish in TCEC, but it might perform unexpectedly well.

[cdani]

Thanks!! I don't know which part of the scaling is due to a good 4cpu implementation and which one is due to good scaling by the search itself, that of course goes deeper on 4cpu.

[Thomas Lagershausen]

Oh sorry. I didn´t see that the cegt-list with 40/120 was only with one core.

But another list is confirming your observation.

https://ssdf.bosjo.net/list.htm

This is a 8-core list and Shredder is here a strong opponent for Komodo 11.01.

So we should have a clother look at Shredder in its further development.

[Laskos]

Oh sorry. I didn´t see that the cegt-list with 40/120 was only with one core.

But another list is confirming your observation.

https://ssdf.bosjo.net/list.htm

This is a 8-core list and Shredder is here a strong opponent for Komodo 11.01.

So we should have a clother look at Shredder in its further development.

Wow, they started to use really powerful hardware, on top of that, playing games with two machines ponder on. Shredder here seems really close Komodo. I downloaded the PGN databases and computed with Ordo the ratings, because I don't know what tool they use (apparently something close to EloStat), and Shredder seems indeed close to Komodo, although errors are pretty large. Ordo also saw many unconnected clusters, so ratings might be shaky.

Thanks for the link!

[Ras]

On the other hand, there seems to be a scaling issue in terms of CPUs. From http://www.computerchess.org.uk/ccrl/40 ... t_all.html :

Code: Select all

Deep Shredder 13 64-bit 4CPU 3288    +15    −15
Shredder 13 64-bit           3267    +13    −13

21 Elo from 1 CPU to 4 CPUs.

Stockfish, 63 Elo:

Code: Select all

Stockfish 8 64-bit 4CPU      3485    +14    −14    
Stockfish 8 64-bit           3422     +9     −9

Houdini, 50 Elo:

Code: Select all

Houdini 5.01 64-bit 4CPU     3471    +16    −16
Houdini 5.01 64-bit          3421     +9     −9    

Komodo, 50 Elo:

Code: Select all

Komodo 11.01 64-bit 4CPU     3461    +19    −19
Komodo 11.01 64-bit          3411    +19    −19

Fire, 80 Elo:

Code: Select all

Fire 5 64-bit 4CPU           3328    +15    −16    
Fire 5 64-bit                3248    +17    −17

On average, these 4 competitors gain 61 Elo while Shredder gains only about a third of that.

[carldaman]

On the other hand, there seems to be a scaling issue in terms of CPUs. From http://www.computerchess.org.uk/ccrl/40 ... t_all.html :

Code: Select all

Deep Shredder 13 64-bit 4CPU 3288    +15    −15
Shredder 13 64-bit           3267    +13    −13

21 Elo from 1 CPU to 4 CPUs.

Stockfish, 63 Elo:

Code: Select all

Stockfish 8 64-bit 4CPU      3485    +14    −14    
Stockfish 8 64-bit           3422     +9     −9

Houdini, 50 Elo:

Code: Select all

Houdini 5.01 64-bit 4CPU     3471    +16    −16
Houdini 5.01 64-bit          3421     +9     −9    

Komodo, 50 Elo:

Code: Select all

Komodo 11.01 64-bit 4CPU     3461    +19    −19
Komodo 11.01 64-bit          3411    +19    −19

Fire, 80 Elo:

Code: Select all

Fire 5 64-bit 4CPU           3328    +15    −16    
Fire 5 64-bit                3248    +17    −17

On average, these 4 competitors gain 61 Elo while Shredder gains only about a third of that.

OTOH, CCRL 40/40 has

Code: Select all

Deep Shredder 13 64-bit 4CPU	3292	+18	−18	54.4%	
 	Shredder 13 64-bit	      3200	+13	−13	53.9%	

CL

[Laskos]

On the other hand, there seems to be a scaling issue in terms of CPUs. From http://www.computerchess.org.uk/ccrl/40 ... t_all.html :

Code: Select all

Deep Shredder 13 64-bit 4CPU 3288    +15    −15
Shredder 13 64-bit           3267    +13    −13

21 Elo from 1 CPU to 4 CPUs.

Stockfish, 63 Elo:

Code: Select all

Stockfish 8 64-bit 4CPU      3485    +14    −14    
Stockfish 8 64-bit           3422     +9     −9

Houdini, 50 Elo:

Code: Select all

Houdini 5.01 64-bit 4CPU     3471    +16    −16
Houdini 5.01 64-bit          3421     +9     −9    

Komodo, 50 Elo:

Code: Select all

Komodo 11.01 64-bit 4CPU     3461    +19    −19
Komodo 11.01 64-bit          3411    +19    −19

Fire, 80 Elo:

Code: Select all

Fire 5 64-bit 4CPU           3328    +15    −16    
Fire 5 64-bit                3248    +17    −17

On average, these 4 competitors gain 61 Elo while Shredder gains only about a third of that.

OTOH, CCRL 40/40 has

Code: Select all

Deep Shredder 13 64-bit 4CPU	3292	+18	−18	54.4%	
 	Shredder 13 64-bit	      3200	+13	−13	53.9%	

CL

There several possible reasons. One, of course is that Shredder behaves this way. Second, there may be statistical flukes. Say, 40/4 4CPU Shredder is deflated in rating by say 15 ELO points, 40/40 4CPU Shredder is inflated by 15 ELO points. Will still scale well, but not by a whopping quantity. Also, new SSDF Thomas linked me to, seems to indicate the same excellent scaling. SSDF started to use very impressive 8 core Ryzen hardware in usually impressive conditions, I downloaded their PGN database and computed the ratings with Ordo. I don't know what rating tool they are using, if it takes care of isolated clusters of engines, also the ratings seem compressed by about 20% (similar to EloStat). Here are the Ordo results for first 20 engines:

Code: Select all

   # PLAYER                           : RATING  ERROR    POINTS  PLAYED     (%)   CFS(next)

   1 Deep Shredder 13 1800X           : 3559.6   47.1     309.5     440    70.3      53    
   2 Komodo 11.01 MP 1800X            : 3557.7   54.9     209.0     294    71.1      95    
   3 Komodo 9.1 MP Q6600              : 3527.5   54.6     650.5     856    76.0      99    
   4 Stockfish 6 MP Q6600             : 3488.3   46.9     663.0     896    74.0      93    
   5 Deep Shredder 13 Q6600           : 3452.5   62.2     164.0     242    67.8      73    
   6 Komodo 7 MP Q6600                : 3437.3   46.2     350.0     572    61.2     100    
   7 Komodo 5.1 MP Q6600              : 3386.3   50.5     525.0     854    61.5      95    
   8 Deep Hiarcs 14 1800X             : 3351.1   56.8     119.5     280    42.7      50    
   9 Stockfish 3 MP Q6600             : 3350.9   43.2     632.0    1069    59.1      80    
  10 Deep Rybka 4 Q6600               : 3340.3   50.7     537.5     894    60.1      70    
  11 Deep Hiarcs 14 Q6600             : 3334.2   35.3     722.0    1194    60.5      89    
  12 Deep Rybka 3 Q6600               : 3321.3   47.1     753.0    1056    71.3      67    
  13 Chiron 3.01 MP Q6600             : 3304.6   83.7      43.0     124    34.7      84    
  14 Naum 4.2 MP Q6600                : 3269.9   48.5     612.5    1031    59.4      94    
  15 Deep Junior Yokohama Q6600       : 3248.4   45.6     293.5     648    45.3      70    
  16 Deep Junior 13.3 Q6600           : 3242.0   40.8     561.5    1115    50.4      69    
  17 Naum 4 MP Q6600                  : 3237.4   39.3     732.5    1187    61.7      52    
  18 Hiarcs 14 A1200                  : 3236.6   47.8     273.0     440    62.0      76    
  19 Spike 1.4 MP Q6600               : 3227.3   43.4     746.0    1363    54.7      53    
  20 Deep Shredder 12 Q6600           : 3226.4   45.3     425.0     832    51.1      52    

This is the closest what we have to TCEC conditions, and although error margins are large, Shredder, if participates in TCEC, might perform well, probably though not quite winning TCEC.

[Modern Times]

Don't forget SSDF use own books as well. Shredder has it's own excellent book. Komodo doesn't have one, so they would use a generic short book of some sort. So you are seeing book performance there as well as engine performance.