Performance of Syzygy and Scorpio
Moderators: hgm, Rebel, chrisw
-
- Posts: 4185
- Joined: Tue Mar 14, 2006 11:34 am
- Location: Ethiopia
Re: Performance of Syzygy and Scorpio
I will check about new Stockfish-egbb implementation but what I changed is to make it pre-load all 5-men at start up. You should see SF usage rise to 400mb at startup. This I did because Adam's results improved with cache and 256mb got very closer to default stockfish that it can bypass it when all 211mb are loaded in RAM. Maybe I messed up there, but what I don't understand is your new Syzygy test result. Previously you said the new additon (probe after pawn pushes) didn't perform better and now they suddenly improved so significantly...
-
- Posts: 10948
- Joined: Wed Jul 26, 2006 10:21 pm
- Full name: Kai Laskos
Re: Performance of Syzygy and Scorpio
That Syzygy WDL improvement was on 6-7 men starting positions with 5-men bases, where we all agree that WDL will help. Yes, SF Scorpio consumed about 400MB of RAM at start-up.Daniel Shawul wrote:I will check about new Stockfish-egbb implementation but what I changed is to make it pre-load all 5-men at start up. You should see SF usage rise to 400mb at startup. This I did because Adam's results improved with cache and 256mb got very closer to default stockfish that it can bypass it when all 211mb are loaded in RAM. Maybe I messed up there, but what I don't understand is your new Syzygy test result. Previously you said the new additon (probe after pawn pushes) didn't perform better and now they suddenly improved so significantly...
-
- Posts: 4185
- Joined: Tue Mar 14, 2006 11:34 am
- Location: Ethiopia
Re: Performance of Syzygy and Scorpio
I don't belive you would get such a performance boost for Syzygy by going from 5-men to 6-men, while at the same time loosing a big performance loosing by a big margin from 6-men to 5-men for Scorpio ? That is why I said you are biased and still don't see why that would be. Maybe I am wrong, but you need to do test games like you did with Houdini with and without bitbases from the start position.Laskos wrote:That Syzygy WDL improvement was on 6-7 men starting positions with 5-men bases, where we all agree that WDL will help. Yes, SF Scorpio consumed about 400MB of RAM at start-up.Daniel Shawul wrote:I will check about new Stockfish-egbb implementation but what I changed is to make it pre-load all 5-men at start up. You should see SF usage rise to 400mb at startup. This I did because Adam's results improved with cache and 256mb got very closer to default stockfish that it can bypass it when all 211mb are loaded in RAM. Maybe I messed up there, but what I don't understand is your new Syzygy test result. Previously you said the new additon (probe after pawn pushes) didn't perform better and now they suddenly improved so significantly...
-
- Posts: 10948
- Joined: Wed Jul 26, 2006 10:21 pm
- Full name: Kai Laskos
Re: Performance of Syzygy and Scorpio
Yes, from normal opening positions would be good, but needs many games, the differences are not that big. Maybe I will do it.Daniel Shawul wrote:I don't belive you would get such a performance boost for Syzygy by going from 5-men to 6-men, while at the same time loosing a big performance loosing by a big margin from 6-men to 5-men for Scorpio ? That is why I said you are biased and still don't see why that would be. Maybe I am wrong, but you need to do test games like you did with Houdini with and without bitbases from the start position.Laskos wrote:That Syzygy WDL improvement was on 6-7 men starting positions with 5-men bases, where we all agree that WDL will help. Yes, SF Scorpio consumed about 400MB of RAM at start-up.Daniel Shawul wrote:I will check about new Stockfish-egbb implementation but what I changed is to make it pre-load all 5-men at start up. You should see SF usage rise to 400mb at startup. This I did because Adam's results improved with cache and 256mb got very closer to default stockfish that it can bypass it when all 211mb are loaded in RAM. Maybe I messed up there, but what I don't understand is your new Syzygy test result. Previously you said the new additon (probe after pawn pushes) didn't perform better and now they suddenly improved so significantly...
-
- Posts: 4185
- Joined: Tue Mar 14, 2006 11:34 am
- Location: Ethiopia
Re: Performance of Syzygy and Scorpio
You can do test games with/without bitbases using something like MES.EPD that have all sorts of endgames (4-20? pieces) with and without bitbases. The positions should be picked randomly since at the start you have very easy positions.Laskos wrote:Yes, from normal opening positions would be good, but needs many games, the differences are not that big. Maybe I will do it.Daniel Shawul wrote:I don't belive you would get such a performance boost for Syzygy by going from 5-men to 6-men, while at the same time loosing a big performance loosing by a big margin from 6-men to 5-men for Scorpio ? That is why I said you are biased and still don't see why that would be. Maybe I am wrong, but you need to do test games like you did with Houdini with and without bitbases from the start position.Laskos wrote:That Syzygy WDL improvement was on 6-7 men starting positions with 5-men bases, where we all agree that WDL will help. Yes, SF Scorpio consumed about 400MB of RAM at start-up.Daniel Shawul wrote:I will check about new Stockfish-egbb implementation but what I changed is to make it pre-load all 5-men at start up. You should see SF usage rise to 400mb at startup. This I did because Adam's results improved with cache and 256mb got very closer to default stockfish that it can bypass it when all 211mb are loaded in RAM. Maybe I messed up there, but what I don't understand is your new Syzygy test result. Previously you said the new additon (probe after pawn pushes) didn't perform better and now they suddenly improved so significantly...
-
- Posts: 3226
- Joined: Wed May 06, 2009 10:31 pm
- Location: Fuquay-Varina, North Carolina
Re: Performance of Syzygy and Scorpio
The copy of egbbso64.so in the zip file did not work for me, but I was able to compile it from the source. The 5 men bitbases definitely load much faster nowDaniel Shawul wrote:Here are binaries of Stockfish and Scorpio for linux-x64. The new egbbso64.so loads all 5 men in less than a second compared to the previous 110 seconds! So default for both is to load all 5 men at startup which take up only 211mb. I guess a 32mb cache should suffice for both after that.
https://dl.dropboxusercontent.com/u/552 ... ngines.zip
Also windows x64 binaries and dlls are included. Copy and replace the three egbbdll files to the directory you have the bitbases. If any of them didn't work for you, all the three git hub repos are updated with the sources so you can compile them yourself.
I am working on creating a set of positions that hopefully will be better suited for testing Scorpio bitbases and Syzygy WDL bases (with Stockfish) than the 3, 4, and 5 men positions I used before.
-
- Posts: 4185
- Joined: Tue Mar 14, 2006 11:34 am
- Location: Ethiopia
Re: Performance of Syzygy and Scorpio
That is good news because I was still not sure if something else was playing a part. Some other people (other than Kai) are reporting problems with them, atleast I know they load fast on one other machine than mine. I have also fixed a display issue with tbhits just now and uploaded the git. I forgot to put a space between the nodes display and 'tbhits' string.Adam Hair wrote:The copy of egbbso64.so in the zip file did not work for me, but I was able to compile it from the source. The 5 men bitbases definitely load much faster nowDaniel Shawul wrote:Here are binaries of Stockfish and Scorpio for linux-x64. The new egbbso64.so loads all 5 men in less than a second compared to the previous 110 seconds! So default for both is to load all 5 men at startup which take up only 211mb. I guess a 32mb cache should suffice for both after that.
https://dl.dropboxusercontent.com/u/552 ... ngines.zip
Also windows x64 binaries and dlls are included. Copy and replace the three egbbdll files to the directory you have the bitbases. If any of them didn't work for you, all the three git hub repos are updated with the sources so you can compile them yourself.
I am working on creating a set of positions that hopefully will be better suited for testing Scorpio bitbases and Syzygy WDL bases (with Stockfish) than the 3, 4, and 5 men positions I used before.
Can you report how many tbhits scorpio and syzygy get? I realize now that syzygy probes too far from the root (6) while scorpio can hit it in qsearch too. I removed that now because stockfish is probably too fast and the last 7/8 plies probably mean nothing for it.
I will make new compiles when you have finished compiling the test set and ready to test.
-
- Posts: 3226
- Joined: Wed May 06, 2009 10:31 pm
- Location: Fuquay-Varina, North Carolina
Re: Performance of Syzygy and Scorpio
When I run the test, I will create logs so that the tbhits info is recorded. It may be next weekend before I am ready to run the tests, but I definitely let you and Ronald know before I start.Daniel Shawul wrote:That is good news because I was still not sure if something else was playing a part. Some other people (other than Kai) are reporting problems with them, atleast I know they load fast on one other machine than mine. I have also fixed a display issue with tbhits just now and uploaded the git. I forgot to put a space between the nodes display and 'tbhits' string.Adam Hair wrote:The copy of egbbso64.so in the zip file did not work for me, but I was able to compile it from the source. The 5 men bitbases definitely load much faster nowDaniel Shawul wrote:Here are binaries of Stockfish and Scorpio for linux-x64. The new egbbso64.so loads all 5 men in less than a second compared to the previous 110 seconds! So default for both is to load all 5 men at startup which take up only 211mb. I guess a 32mb cache should suffice for both after that.
https://dl.dropboxusercontent.com/u/552 ... ngines.zip
Also windows x64 binaries and dlls are included. Copy and replace the three egbbdll files to the directory you have the bitbases. If any of them didn't work for you, all the three git hub repos are updated with the sources so you can compile them yourself.
I am working on creating a set of positions that hopefully will be better suited for testing Scorpio bitbases and Syzygy WDL bases (with Stockfish) than the 3, 4, and 5 men positions I used before.
Can you report how many tbhits scorpio and syzygy get? I realize now that syzygy probes too far from the root (6) while scorpio can hit it in qsearch too. I removed that now because stockfish is probably too fast and the last 7/8 plies probably mean nothing for it.
I will make new compiles when you have finished compiling the test set and ready to test.
-
- Posts: 10948
- Joined: Wed Jul 26, 2006 10:21 pm
- Full name: Kai Laskos
Re: Performance of Syzygy and Scorpio
I did this test with MES.EPD at 10'' + 0.1''Daniel Shawul wrote: You can do test games with/without bitbases using something like MES.EPD that have all sorts of endgames (4-20? pieces) with and without bitbases. The positions should be picked randomly since at the start you have very easy positions.
Code: Select all
10'' + 0.1''
Program Score % Elo + - Draws
1 SF 05.02 Syzygy : 3109.5/6000 51.8 10 7 7 41.8 %
2 SF 05.02 Syzygy WDL : 3019.0/6000 50.3 2 7 7 42.6 %
3 SF 05.02 NO Syzygy : 2981.0/6000 49.7 -2 7 7 42.9 %
4 SF Scorpio : 2957.0/6000 49.3 -4 7 7 43.1 %
5 SF NO Scorpio : 2933.5/6000 48.9 -6 7 7 43.3 %
-
- Posts: 4185
- Joined: Tue Mar 14, 2006 11:34 am
- Location: Ethiopia
Re: Performance of Syzygy and Scorpio
Well I already mentioned to Adam syzygy is probed much less frequently than I did because it probes 6 plies away from the root, while mine can do it in qsearch 0 plies. If I used 6 plies, my nps more than doubles but that doesn't work for scorpio which reaches much less depth and no endgame knowledge. I will fix it for Stockfish ofcourse.To note that SF Scorpio had a dramatic loss in NPS, on average by a factor 1.7 or so.
Code: Select all
1 SF 05.02 Syzygy : 3109.5/6000 51.8 10 7 7 41.8 %
2 SF 05.02 Syzygy WDL : 3019.0/6000 50.3 2 7 7 42.6 %
3 SF 05.02 NO Syzygy : 2981.0/6000 49.7 -2 7 7 42.9 %
4 SF Scorpio : 2957.0/6000 49.3 -4 7 7 43.1 %
5 SF NO Scorpio : 2933.5/6000 48.9 -6 7 7 43.3 %
Code: Select all
1 SF 05.02 Syzygy : 2071.0/4000
2 SF 05.02 Syzygy WDL : 2042.0/4000
3 SF NO Scorpio : 1978.5/4000
4 SF 05.02 NO Syzygy : 1966.5/4000
5 SF Scorpio : 1942.0/4000