Performance of Syzygy and Scorpio

[Daniel Shawul]

Here are binaries of Stockfish and Scorpio for linux-x64. The new egbbso64.so loads all 5 men in less than a second compared to the previous 110 seconds! So default for both is to load all 5 men at startup which take up only 211mb. I guess a 32mb cache should suffice for both after that.

https://dl.dropboxusercontent.com/u/552 ... ngines.zip

Also windows x64 binaries and dlls are included. Copy and replace the three egbbdll files to the directory you have the bitbases. If any of them didn't work for you, all the three git hub repos are updated with the sources so you can compile them yourself.

[Houdini]

It maybe very hard to find positions where Stockfish search can not see a KNOWN_WIN from the root. It just has a lot of internal node recognizers for the common cases. The best measurement would be to find a weaker engine since we still need perfect play by DTM players.

Finally you understand that it takes a rather weak engine without a lot of endgame knowledge for your "ensure progress" code to have any usefulness.
Houdini, like Stockfish, does not qualify for this - strong engines require DTM/DTZ information to really improve their play.

I see that you have toned down your language considerable compared to a couple of pages ago...

[Adam Hair]

Here are binaries of Stockfish and Scorpio for linux-x64. The new egbbso64.so loads all 5 men in less than a second compared to the previous 110 seconds! So default for both is to load all 5 men at startup which take up only 211mb. I guess a 32mb cache should suffice for both after that.

https://dl.dropboxusercontent.com/u/552 ... ngines.zip

Also windows x64 binaries and dlls are included. Copy and replace the three egbbdll files to the directory you have the bitbases. If any of them didn't work for you, all the three git hub repos are updated with the sources so you can compile them yourself.

I will work on a new test over the weekend.

[Ryan Benitez]

It maybe very hard to find positions where Stockfish search can not see a KNOWN_WIN from the root. It just has a lot of internal node recognizers for the common cases. The best measurement would be to find a weaker engine since we still need perfect play by DTM players.

Finally you understand that it takes a rather weak engine without a lot of endgame knowledge for your "ensure progress" code to have any usefulness.
Houdini, like Stockfish, does not qualify for this - strong engines require DTM/DTZ information to really improve their play.

I see that you have toned down your language considerable compared to a couple of pages ago...

I suspect test conditions are causing confusion on the matter as well. It still seems unlikely that chess is the lone special case where DTM is needed.

[Laskos]

It maybe very hard to find positions where Stockfish search can not see a KNOWN_WIN from the root. It just has a lot of internal node recognizers for the common cases. The best measurement would be to find a weaker engine since we still need perfect play by DTM players.

Finally you understand that it takes a rather weak engine without a lot of endgame knowledge for your "ensure progress" code to have any usefulness.
Houdini, like Stockfish, does not qualify for this - strong engines require DTM/DTZ information to really improve their play.

I see that you have toned down your language considerable compared to a couple of pages ago...

It seems true. If I did the tests correctly (I had some problems with Scorpio engine in Cutechess-Cli), these are the results with the newest Scorpio EGBBs, DLLs and SF supporting them provided by Daniel.
Hard 5-men TB wins (in at least 30-40 moves) suite, engines with 5-men bases, Shredder 12 Nalimov being the perfect player (500/1000 being the perfect score)

Code: Select all

10'' + 0.1''

    Program                            Score  

  1 Shredder 12 Nalimov           : 2820.0/5000
  
  2 SF 05.02 Syzygy                : 500.0/1000  
  3 SF 05.02 NO Syzygy             : 426.0/1000 
  4 SF 05.02 Syzygy WDL            : 424.0/1000  
  5 SF NO Scorpio                  : 424.0/1000  
  6 SF Scorpio                     : 406.0/1000 

Scorpio EGBBs seem to do more harm to SF than good. On the other hand, with the weaker Scorpio engine (the latest provided by Daniel), Scorpio EGBBs improve the play:

Code: Select all

10'' + 0.1''

    Proggram                           Score

  1 Shredder 12 Nalimov           : 1349.0/2000
  
  2 Scorpio EGBB                   : 368.0/1000  
  3 Scorpio NO EGBB                : 283.0/1000 

[Daniel Shawul]

Kai, you seem biased and eager to make conclusions. Adam produced results that says scorpio EGBBs performed but you want to believe fluke results that suit your biased conclusions.

Code: Select all

1 Gaviota GTB                            9   9  6000    52.9 %   2390    5.9 % 
  2 Stockfish Syzygy none                 21  21  1000    47.9 %   2410    4.2 % 
  3 Stockfish Scorpio none                21  21  1000    47.6 %   2410    4.8 % 
  4 Stockfish Scorpio EGBB  cache 256     21  21  1000    47.1 %   2410    5.7 % 
  5 Stockfish Scorpio EGBB cache 16       21  21  1000    46.8 %   2410    6.5 % 
  6 Stockfish Syzygy WDL probe depth 1    21  21  1000    46.5 %   2410    7.1 % 
  7 Stockfish Syzygy WDL probe depth 4    21  21  1000    46.5 %   2410    7.1 % 

So here I 'conclude" Syzyygy suck??
Anyway your test is screwed because Stockfish solves most 5-men from the root by its big-search depth + endgame recongnizers. Making a conclustion from this 5-men test is ridiclous because TBs are probed from the start position while in real game play you probe them at the leaves or qsearch where you don't have the luxury of search depth.

Good to know that both Scorpio/Toga improved game play though. They don't have internal recognizers so they will improve on endgame tests even on this simple 5-men tests. For Stockfish unfortunately the real test is to play games from the start.

[Daniel Shawul]

It maybe very hard to find positions where Stockfish search can not see a KNOWN_WIN from the root. It just has a lot of internal node recognizers for the common cases. The best measurement would be to find a weaker engine since we still need perfect play by DTM players.

Finally you understand that it takes a rather weak engine without a lot of endgame knowledge for your "ensure progress" code to have any usefulness.
Houdini, like Stockfish, does not qualify for this - strong engines require DTM/DTZ information to really improve their play.

I see that you have toned down your language considerable compared to a couple of pages ago...

I suspect test conditions are causing confusion on the matter as well. It still seems unlikely that chess is the lone special case where DTM is needed.

It is ridiclous to say DTM is the real deal. He is just trying to cover up for his screwed up implementation of bitbases.

[Laskos]

Kai, you seem biased and eager to make conclusions. Adam produced results that says scorpio EGBBs performed but you want to believe fluke results that suit your biased conclusions.

Code: Select all

1 Gaviota GTB                            9   9  6000    52.9 %   2390    5.9 % 
  2 Stockfish Syzygy none                 21  21  1000    47.9 %   2410    4.2 % 
  3 Stockfish Scorpio none                21  21  1000    47.6 %   2410    4.8 % 
  4 Stockfish Scorpio EGBB  cache 256     21  21  1000    47.1 %   2410    5.7 % 
  5 Stockfish Scorpio EGBB cache 16       21  21  1000    46.8 %   2410    6.5 % 
  6 Stockfish Syzygy WDL probe depth 1    21  21  1000    46.5 %   2410    7.1 % 
  7 Stockfish Syzygy WDL probe depth 4    21  21  1000    46.5 %   2410    7.1 % 

So here I 'conclude" Syzyygy suck??
Anyway your test is screwed because Stockfish solves most 5-men from the root by its big-search depth + endgame recongnizers. Making a conclustion from this 5-men test is ridiclous because TBs are probed from the start position while in real game play you probe them at the leaves or qsearch where you don't have the luxury of search depth.

Good to know that both Scorpio/Toga improved game play though. They don't have internal recognizers so they will improve on endgame tests even on this simple 5-men tests. For Stockfish unfortunately the real test is to play games from the start.

Adam's SF implementation was the old one, besides that, Adam confirms that neither Scorpio EGBB, nor Syzygy WDL improve Stockfish result from 5-men root positions. I have a test from 6-7 men positions with new 5-men EGBBs (loaded into RAM), which are again not very good for Scorpio EGBBs. Round-Robin:

Code: Select all

    Program                            Score     

  1 SF 05.02 Syzygy                : 2071.0/4000
  2 SF 05.02 Syzygy WDL            : 2042.0/4000
  3 SF NO Scorpio                  : 1978.5/4000 
  4 SF 05.02 NO Syzygy             : 1966.5/4000
  5 SF Scorpio                     : 1942.0/4000

I am not biased, I got these results, it's either me screwing up the tests, or it's you mangling the latest implementation for SF. With the old implementation I got results very similar to Adam's. Let's see Adam's results with the new implementation.

[Daniel Shawul]

It is not SF result that suddenly changed but Scorpio's. Adam or You said the new implementation of syzygy 'progress wdl' had no effect. Which btw is added fragments from scorpio progress egbb by probing after pawn pushes. That is what is the 'new implementation' if you didn't know. Captures/promos account for most progress but there is more to account for... Anyway I don't expect many to understand this.

Edit Infact it is YOU who said it

Here is the result with the latest version, using hard 4-5 men TB wins file. No significant change:

Code: Select all

10'' + 0.1'' 

    Program                            Score      

  1 Shredder 12 Nalimov           : 2228.5/4000 
  
  2 SF 05.02 Syzygy                : 500.0/1000  
  3 SF 05.02 NO TB                 : 429.5/1000  
  4 SF 05.02 Syzygy WDL            : 422.0/1000  
  5 SF Scorpio                     : 420.0/1000 

[Laskos]

It is not SF result that suddenly changed but Scorpio's. Adam or You said the new implementation of syzygy 'progress wdl' had no effect. Which btw is added fragments from scorpio progress egbb by probing after pawn pushes. That is what is the 'new implementation' if you didn't know. Captures/promos account for most progress but there is more to account for... Anyway I don't expect many to understand this.

Edit Infact it is YOU who said it

Here is the result with the latest version, using hard 4-5 men TB wins file. No significant change:

Code: Select all

10'' + 0.1'' 

    Program                            Score      

  1 Shredder 12 Nalimov           : 2228.5/4000 
  
  2 SF 05.02 Syzygy                : 500.0/1000  
  3 SF 05.02 NO TB                 : 429.5/1000  
  4 SF 05.02 Syzygy WDL            : 422.0/1000  
  5 SF Scorpio                     : 420.0/1000 

Yes, and the latest test confirms that, only that Scorpio EGBBs perform even worse (a statistical fluke?):

Code: Select all

10'' + 0.1'' 

     Program                            Score  

   1 Shredder 12 Nalimov           : 2820.0/5000 
   
   2 SF 05.02 Syzygy                : 500.0/1000  
   3 SF 05.02 NO Syzygy             : 426.0/1000 
   4 SF 05.02 Syzygy WDL            : 424.0/1000  
   5 SF NO Scorpio                  : 424.0/1000  
   6 SF Scorpio                     : 406.0/1000 

I saw a big drop in SF NPS with EGBBs, but I am not sure what that means.