Performance of Syzygy and Scorpio

[Daniel Shawul]

I will check about new Stockfish-egbb implementation but what I changed is to make it pre-load all 5-men at start up. You should see SF usage rise to 400mb at startup. This I did because Adam's results improved with cache and 256mb got very closer to default stockfish that it can bypass it when all 211mb are loaded in RAM. Maybe I messed up there, but what I don't understand is your new Syzygy test result. Previously you said the new additon (probe after pawn pushes) didn't perform better and now they suddenly improved so significantly...

[Laskos]

I will check about new Stockfish-egbb implementation but what I changed is to make it pre-load all 5-men at start up. You should see SF usage rise to 400mb at startup. This I did because Adam's results improved with cache and 256mb got very closer to default stockfish that it can bypass it when all 211mb are loaded in RAM. Maybe I messed up there, but what I don't understand is your new Syzygy test result. Previously you said the new additon (probe after pawn pushes) didn't perform better and now they suddenly improved so significantly...

That Syzygy WDL improvement was on 6-7 men starting positions with 5-men bases, where we all agree that WDL will help. Yes, SF Scorpio consumed about 400MB of RAM at start-up.

[Daniel Shawul]

I will check about new Stockfish-egbb implementation but what I changed is to make it pre-load all 5-men at start up. You should see SF usage rise to 400mb at startup. This I did because Adam's results improved with cache and 256mb got very closer to default stockfish that it can bypass it when all 211mb are loaded in RAM. Maybe I messed up there, but what I don't understand is your new Syzygy test result. Previously you said the new additon (probe after pawn pushes) didn't perform better and now they suddenly improved so significantly...

That Syzygy WDL improvement was on 6-7 men starting positions with 5-men bases, where we all agree that WDL will help. Yes, SF Scorpio consumed about 400MB of RAM at start-up.

I don't belive you would get such a performance boost for Syzygy by going from 5-men to 6-men, while at the same time loosing a big performance loosing by a big margin from 6-men to 5-men for Scorpio ? That is why I said you are biased and still don't see why that would be. Maybe I am wrong, but you need to do test games like you did with Houdini with and without bitbases from the start position.

[Laskos]

I will check about new Stockfish-egbb implementation but what I changed is to make it pre-load all 5-men at start up. You should see SF usage rise to 400mb at startup. This I did because Adam's results improved with cache and 256mb got very closer to default stockfish that it can bypass it when all 211mb are loaded in RAM. Maybe I messed up there, but what I don't understand is your new Syzygy test result. Previously you said the new additon (probe after pawn pushes) didn't perform better and now they suddenly improved so significantly...

That Syzygy WDL improvement was on 6-7 men starting positions with 5-men bases, where we all agree that WDL will help. Yes, SF Scorpio consumed about 400MB of RAM at start-up.

I don't belive you would get such a performance boost for Syzygy by going from 5-men to 6-men, while at the same time loosing a big performance loosing by a big margin from 6-men to 5-men for Scorpio ? That is why I said you are biased and still don't see why that would be. Maybe I am wrong, but you need to do test games like you did with Houdini with and without bitbases from the start position.

Yes, from normal opening positions would be good, but needs many games, the differences are not that big. Maybe I will do it.

[Daniel Shawul]

I will check about new Stockfish-egbb implementation but what I changed is to make it pre-load all 5-men at start up. You should see SF usage rise to 400mb at startup. This I did because Adam's results improved with cache and 256mb got very closer to default stockfish that it can bypass it when all 211mb are loaded in RAM. Maybe I messed up there, but what I don't understand is your new Syzygy test result. Previously you said the new additon (probe after pawn pushes) didn't perform better and now they suddenly improved so significantly...

That Syzygy WDL improvement was on 6-7 men starting positions with 5-men bases, where we all agree that WDL will help. Yes, SF Scorpio consumed about 400MB of RAM at start-up.

I don't belive you would get such a performance boost for Syzygy by going from 5-men to 6-men, while at the same time loosing a big performance loosing by a big margin from 6-men to 5-men for Scorpio ? That is why I said you are biased and still don't see why that would be. Maybe I am wrong, but you need to do test games like you did with Houdini with and without bitbases from the start position.

Yes, from normal opening positions would be good, but needs many games, the differences are not that big. Maybe I will do it.

You can do test games with/without bitbases using something like MES.EPD that have all sorts of endgames (4-20? pieces) with and without bitbases. The positions should be picked randomly since at the start you have very easy positions.

[Adam Hair]

Here are binaries of Stockfish and Scorpio for linux-x64. The new egbbso64.so loads all 5 men in less than a second compared to the previous 110 seconds! So default for both is to load all 5 men at startup which take up only 211mb. I guess a 32mb cache should suffice for both after that.

https://dl.dropboxusercontent.com/u/552 ... ngines.zip

Also windows x64 binaries and dlls are included. Copy and replace the three egbbdll files to the directory you have the bitbases. If any of them didn't work for you, all the three git hub repos are updated with the sources so you can compile them yourself.

The copy of egbbso64.so in the zip file did not work for me, but I was able to compile it from the source. The 5 men bitbases definitely load much faster now

I am working on creating a set of positions that hopefully will be better suited for testing Scorpio bitbases and Syzygy WDL bases (with Stockfish) than the 3, 4, and 5 men positions I used before.

[Daniel Shawul]

Here are binaries of Stockfish and Scorpio for linux-x64. The new egbbso64.so loads all 5 men in less than a second compared to the previous 110 seconds! So default for both is to load all 5 men at startup which take up only 211mb. I guess a 32mb cache should suffice for both after that.

https://dl.dropboxusercontent.com/u/552 ... ngines.zip

Also windows x64 binaries and dlls are included. Copy and replace the three egbbdll files to the directory you have the bitbases. If any of them didn't work for you, all the three git hub repos are updated with the sources so you can compile them yourself.

The copy of egbbso64.so in the zip file did not work for me, but I was able to compile it from the source. The 5 men bitbases definitely load much faster now

I am working on creating a set of positions that hopefully will be better suited for testing Scorpio bitbases and Syzygy WDL bases (with Stockfish) than the 3, 4, and 5 men positions I used before.

That is good news because I was still not sure if something else was playing a part. Some other people (other than Kai) are reporting problems with them, atleast I know they load fast on one other machine than mine. I have also fixed a display issue with tbhits just now and uploaded the git. I forgot to put a space between the nodes display and 'tbhits' string.

Can you report how many tbhits scorpio and syzygy get? I realize now that syzygy probes too far from the root (6) while scorpio can hit it in qsearch too. I removed that now because stockfish is probably too fast and the last 7/8 plies probably mean nothing for it.

I will make new compiles when you have finished compiling the test set and ready to test.

[Adam Hair]

Here are binaries of Stockfish and Scorpio for linux-x64. The new egbbso64.so loads all 5 men in less than a second compared to the previous 110 seconds! So default for both is to load all 5 men at startup which take up only 211mb. I guess a 32mb cache should suffice for both after that.

https://dl.dropboxusercontent.com/u/552 ... ngines.zip

Also windows x64 binaries and dlls are included. Copy and replace the three egbbdll files to the directory you have the bitbases. If any of them didn't work for you, all the three git hub repos are updated with the sources so you can compile them yourself.

The copy of egbbso64.so in the zip file did not work for me, but I was able to compile it from the source. The 5 men bitbases definitely load much faster now

I am working on creating a set of positions that hopefully will be better suited for testing Scorpio bitbases and Syzygy WDL bases (with Stockfish) than the 3, 4, and 5 men positions I used before.

That is good news because I was still not sure if something else was playing a part. Some other people (other than Kai) are reporting problems with them, atleast I know they load fast on one other machine than mine. I have also fixed a display issue with tbhits just now and uploaded the git. I forgot to put a space between the nodes display and 'tbhits' string.

Can you report how many tbhits scorpio and syzygy get? I realize now that syzygy probes too far from the root (6) while scorpio can hit it in qsearch too. I removed that now because stockfish is probably too fast and the last 7/8 plies probably mean nothing for it.

I will make new compiles when you have finished compiling the test set and ready to test.

When I run the test, I will create logs so that the tbhits info is recorded. It may be next weekend before I am ready to run the tests, but I definitely let you and Ronald know before I start.

[Laskos]

You can do test games with/without bitbases using something like MES.EPD that have all sorts of endgames (4-20? pieces) with and without bitbases. The positions should be picked randomly since at the start you have very easy positions.

I did this test with MES.EPD at 10'' + 0.1''

Code: Select all

10'' + 0.1''

    Program                            Score     %      Elo    +   -    Draws

  1 SF 05.02 Syzygy                : 3109.5/6000  51.8   10    7   7   41.8 %
  2 SF 05.02 Syzygy WDL            : 3019.0/6000  50.3    2    7   7   42.6 %
  3 SF 05.02 NO Syzygy             : 2981.0/6000  49.7   -2    7   7   42.9 %
  4 SF Scorpio                     : 2957.0/6000  49.3   -4    7   7   43.1 %
  5 SF NO Scorpio                  : 2933.5/6000  48.9   -6    7   7   43.3 %

To note that SF Scorpio had a dramatic loss in NPS, on average by a factor 1.7 or so.

[Daniel Shawul]

To note that SF Scorpio had a dramatic loss in NPS, on average by a factor 1.7 or so.

Well I already mentioned to Adam syzygy is probed much less frequently than I did because it probes 6 plies away from the root, while mine can do it in qsearch 0 plies. If I used 6 plies, my nps more than doubles but that doesn't work for scorpio which reaches much less depth and no endgame knowledge. I will fix it for Stockfish ofcourse.

Code: Select all

1 SF 05.02 Syzygy                : 3109.5/6000  51.8   10    7   7   41.8 % 
  2 SF 05.02 Syzygy WDL            : 3019.0/6000  50.3    2    7   7   42.6 % 
  3 SF 05.02 NO Syzygy             : 2981.0/6000  49.7   -2    7   7   42.9 % 
  4 SF Scorpio                     : 2957.0/6000  49.3   -4    7   7   43.1 % 
  5 SF NO Scorpio                  : 2933.5/6000  48.9   -6    7   7   43.3 % 

The "SF 05.02 NO Syzygy" scored more than 50 points compared to "SF NO Scorpio" to begin with, so it is hard to compare but atleast now Scorpio EGBBs show as much improvement as syzygy WDL. Also why does "NO syzygy" version scores much better here than in the previous case anyway ?

Code: Select all

 1 SF 05.02 Syzygy                : 2071.0/4000 
  2 SF 05.02 Syzygy WDL            : 2042.0/4000 
  3 SF NO Scorpio                  : 1978.5/4000 
  4 SF 05.02 NO Syzygy             : 1966.5/4000 
  5 SF Scorpio                     : 1942.0/4000 

Another significant difference is Syzygy WDL also scored significantly less than in the previous test. The updated version of Syzygy WDL that probes right after (captures/pawns i.e. fifty=0) is an improvement but it murkies the comparison because now it is using progress making code, so I prefered you tested the older version. Measuring the other differences with KBNK , KBBKN wins that don't involve captures or pawn pushes is very difficult because MES or other gerneral EPD do not have those a lot. I believe here we are just comparing nps differences due to probe settings more than anything else.