Performance of Syzygy and Scorpio

Discussion of anything and everything relating to chess playing software and machines.

Moderators: hgm, Rebel, chrisw

User avatar
Laskos
Posts: 10948
Joined: Wed Jul 26, 2006 10:21 pm
Full name: Kai Laskos

Performance of Syzygy and Scorpio

Post by Laskos »

I ran two gauntlets, at 10''+0.1'' and 20''+0.2'' from Wins.epd, an EPD file which contains 3-4-5 men TB wins, so the TB positions are at the root. I used 3-4-5 men Nalimovs, Syzygy and Scorpio (old ones). The gauntlet was against Houdini Nalimov, which I assumed to be a perfect player in these conditions (aside from 50-move rule). Default 32MB cache size for Nalimovs and the default 16MB Scorpio cache size. The perfect score would be 500/1000, as the positions are reversed, anything lower than 500 shows the inaccuracy of the Endgame Bases at the root, i.e. failure to convert a TB win.

Code: Select all

10'' + 0.1''

    Program                            Score     

  1 Houdini 4 Nalimov             : 2123.5/4000

  2 Houdini 4 Syzygy               : 500.0/1000  
  3 Houdini 4 NO TB                : 481.0/1000  
  4 Toga Scorpio                   : 470.5/1000  
  5 Toga NO TB                     : 425.0/1000 



20'' + 0.2''

    Program                            Score     

  1 Houdini 4 Nalimov             : 2138.5/4000
 
  2 Houdini 4 Syzygy               : 500.0/1000 
  3 Toga Scorpio                   : 471.5/1000 
  4 Houdini 4 NO TB                : 469.0/1000 
  5 Toga NO TB                     : 421.0/1000 
User avatar
velmarin
Posts: 1600
Joined: Mon Feb 21, 2011 9:48 am

Re: Performance of Syzygy and Scorpio

Post by velmarin »

It is an idea on my mind.
I thought of a confrontation between "Stockfish Syzygy" and "Stockfish Scorpio"l, both with the same code Stockfish.
But I wanted to do it with some collection of FEN with more pieces, since it is supposed to be pretty bitbases consulted before the end.

Thanks Kai.
Jose.
Daniel Shawul
Posts: 4185
Joined: Tue Mar 14, 2006 11:34 am
Location: Ethiopia

Re: Performance of Syzygy and Scorpio

Post by Daniel Shawul »

Toga's implementation of EGBBs is exemplary compared to you know who 's :)
That is an improvement of about 12% more solutions found. Stockfish has now also the super compact egbbs FYI.
User avatar
Houdini
Posts: 1471
Joined: Tue Mar 16, 2010 12:00 am

Re: Performance of Syzygy and Scorpio

Post by Houdini »

Interesting albeit unsurprising results:
- The Houdini Syzygy implementation produces a perfect 500/1000 score.
- Toga's "exemplary" egbb implementation only scores 470/1000.

This confirms the obvious: with egbb (or any other WDL bases) engines fail to convert some winning positions. WDL bases simply do not contain the winning path, and no piece of code can provide a work-around for this shortcoming.

For a reliable conversion of winning positions end game bases with DTW or DTZ information are required.

This also illustrates why we do not recommend using Houdini + Syzygy bases without the DTZ files.
syzygy
Posts: 5555
Joined: Tue Feb 28, 2012 11:56 pm

Re: Performance of Syzygy and Scorpio

Post by syzygy »

It would be interesting to see the result for SF+syzygy with only the WDL tables (just leave the DTZ tables out of the path, assuming you have them in a separate directory).
Daniel Shawul
Posts: 4185
Joined: Tue Mar 14, 2006 11:34 am
Location: Ethiopia

Re: Performance of Syzygy and Scorpio

Post by Daniel Shawul »

Yep, Nalimov TBs play perfect chess but that is not what you need, because it doesn't translate to strength. Last time Kai did a test was run with Nalimov + Shredder bitbases it didn't improve it by much if at all. Bitbases are all that matter and not that 100s of gigabytes of wasted space that is probed only at the root.
User avatar
Houdini
Posts: 1471
Joined: Tue Mar 16, 2010 12:00 am

Re: Performance of Syzygy and Scorpio

Post by Houdini »

syzygy wrote:It would be interesting to see the result for SF+syzygy with only the WDL tables (just leave the DTZ tables out of the path, assuming you have them in a separate directory).
I've never tested Houdini 4+Syzyy without DTZ tables, it's an interesting test but not recommended for game play.
User avatar
michiguel
Posts: 6401
Joined: Thu Mar 09, 2006 8:30 pm
Location: Chicago, Illinois, USA

Re: Performance of Syzygy and Scorpio

Post by michiguel »

[MODERATION]
Thread temporarily locked until we look at it (don't have time now). Anyway, maybe few hours of lock may help to calm down the spirits, before there is an escalation.

Miguel

[EDIT]
A huge branch with mostly a rehash of an old fight was removed. If there is anything of value that I miss, please repost, but I did not see it. Generally a bit of that is ok, but this was drowning the thread. So, let's move on.
User avatar
Laskos
Posts: 10948
Joined: Wed Jul 26, 2006 10:21 pm
Full name: Kai Laskos

Re: Performance of Syzygy and Scorpio

Post by Laskos »

syzygy wrote:It would be interesting to see the result for SF+syzygy with only the WDL tables (just leave the DTZ tables out of the path, assuming you have them in a separate directory).
I used an EPD file with 3-4-5 men wins, having a bit harder positions, wins in at least 30-40 moves. Shredder 12 with Nalimovs is the perfect player here:

Code: Select all

10'' + 0.1''

    Program                            Score 

  1 Shredder 12 Nalimov           : 2911.0/5000 

  2 SF 04.02 Syzygy                : 500.0/1000
  3 SF 04.02 NO TB                 : 422.5/1000
  4 SF 04.02 Syzygy WDL            : 419.0/1000
  5 Toga Scorpio                   : 415.5/1000
  6 Toga NO TB                     : 332.0/1000
Syzygy WDL doesn't help SF at all.
User avatar
M ANSARI
Posts: 3707
Joined: Thu Mar 16, 2006 7:10 pm

Re: Performance of Syzygy and Scorpio

Post by M ANSARI »

I cannot imagine any EGTB without DTZ would be of help. The engine simply doesn't have the capacity to understand many wins as the wins require moves which if "fixed" in the algorithmic sense ... they would make the engine play poorly in the majority of other positions. Somehow I feel that there are some low lying fruit in easy ELO points gained by using a different scheme than EGTB's in chess. Of course SSD's are a huge boost and the dramatic compression that bitbases have made possible is also tremendous. But if you look at chess endgames it sometimes feels like quantum mechanics ... things at the minute level do not conform to normal physics laws. Maybe a normal CPU chess algo is not the correct way to deal with endgames and the solution lies with a daughter card (like a VGA card with many CUDA processors) that can do EGTB's on the fly and can see "no progress" and find "progress" using little chess knowledge and more Monte Carlo probing also on the fly. The abilities of the latest VGA cards is really phenomenal ... it would seem that a combination of fast memory RAM or SSD and fast CUDA processing could be the better path to take for endgames rather than just have a giant storage bank of EGTB's getting probed by a normal chess engine using an integer based CPU algo.