Page 1 of 11

Performance of Syzygy and Scorpio

Posted: Tue Feb 04, 2014 11:55 am
by Laskos
I ran two gauntlets, at 10''+0.1'' and 20''+0.2'' from Wins.epd, an EPD file which contains 3-4-5 men TB wins, so the TB positions are at the root. I used 3-4-5 men Nalimovs, Syzygy and Scorpio (old ones). The gauntlet was against Houdini Nalimov, which I assumed to be a perfect player in these conditions (aside from 50-move rule). Default 32MB cache size for Nalimovs and the default 16MB Scorpio cache size. The perfect score would be 500/1000, as the positions are reversed, anything lower than 500 shows the inaccuracy of the Endgame Bases at the root, i.e. failure to convert a TB win.

Code: Select all

10'' + 0.1''

    Program                            Score     

  1 Houdini 4 Nalimov             : 2123.5/4000

  2 Houdini 4 Syzygy               : 500.0/1000  
  3 Houdini 4 NO TB                : 481.0/1000  
  4 Toga Scorpio                   : 470.5/1000  
  5 Toga NO TB                     : 425.0/1000 



20'' + 0.2''

    Program                            Score     

  1 Houdini 4 Nalimov             : 2138.5/4000
 
  2 Houdini 4 Syzygy               : 500.0/1000 
  3 Toga Scorpio                   : 471.5/1000 
  4 Houdini 4 NO TB                : 469.0/1000 
  5 Toga NO TB                     : 421.0/1000 

Re: Performance of Syzygy and Scorpio

Posted: Tue Feb 04, 2014 1:37 pm
by velmarin
It is an idea on my mind.
I thought of a confrontation between "Stockfish Syzygy" and "Stockfish Scorpio"l, both with the same code Stockfish.
But I wanted to do it with some collection of FEN with more pieces, since it is supposed to be pretty bitbases consulted before the end.

Thanks Kai.
Jose.

Re: Performance of Syzygy and Scorpio

Posted: Tue Feb 04, 2014 3:39 pm
by Daniel Shawul
Toga's implementation of EGBBs is exemplary compared to you know who 's :)
That is an improvement of about 12% more solutions found. Stockfish has now also the super compact egbbs FYI.

Re: Performance of Syzygy and Scorpio

Posted: Tue Feb 04, 2014 4:17 pm
by Houdini
Interesting albeit unsurprising results:
- The Houdini Syzygy implementation produces a perfect 500/1000 score.
- Toga's "exemplary" egbb implementation only scores 470/1000.

This confirms the obvious: with egbb (or any other WDL bases) engines fail to convert some winning positions. WDL bases simply do not contain the winning path, and no piece of code can provide a work-around for this shortcoming.

For a reliable conversion of winning positions end game bases with DTW or DTZ information are required.

This also illustrates why we do not recommend using Houdini + Syzygy bases without the DTZ files.

Re: Performance of Syzygy and Scorpio

Posted: Tue Feb 04, 2014 4:49 pm
by syzygy
It would be interesting to see the result for SF+syzygy with only the WDL tables (just leave the DTZ tables out of the path, assuming you have them in a separate directory).

Re: Performance of Syzygy and Scorpio

Posted: Tue Feb 04, 2014 4:51 pm
by Daniel Shawul
Yep, Nalimov TBs play perfect chess but that is not what you need, because it doesn't translate to strength. Last time Kai did a test was run with Nalimov + Shredder bitbases it didn't improve it by much if at all. Bitbases are all that matter and not that 100s of gigabytes of wasted space that is probed only at the root.

Re: Performance of Syzygy and Scorpio

Posted: Tue Feb 04, 2014 5:05 pm
by Houdini
syzygy wrote:It would be interesting to see the result for SF+syzygy with only the WDL tables (just leave the DTZ tables out of the path, assuming you have them in a separate directory).
I've never tested Houdini 4+Syzyy without DTZ tables, it's an interesting test but not recommended for game play.

Re: Performance of Syzygy and Scorpio

Posted: Tue Feb 04, 2014 6:08 pm
by michiguel
[MODERATION]
Thread temporarily locked until we look at it (don't have time now). Anyway, maybe few hours of lock may help to calm down the spirits, before there is an escalation.

Miguel

[EDIT]
A huge branch with mostly a rehash of an old fight was removed. If there is anything of value that I miss, please repost, but I did not see it. Generally a bit of that is ok, but this was drowning the thread. So, let's move on.

Re: Performance of Syzygy and Scorpio

Posted: Wed Feb 05, 2014 9:38 am
by Laskos
syzygy wrote:It would be interesting to see the result for SF+syzygy with only the WDL tables (just leave the DTZ tables out of the path, assuming you have them in a separate directory).
I used an EPD file with 3-4-5 men wins, having a bit harder positions, wins in at least 30-40 moves. Shredder 12 with Nalimovs is the perfect player here:

Code: Select all

10'' + 0.1''

    Program                            Score 

  1 Shredder 12 Nalimov           : 2911.0/5000 

  2 SF 04.02 Syzygy                : 500.0/1000
  3 SF 04.02 NO TB                 : 422.5/1000
  4 SF 04.02 Syzygy WDL            : 419.0/1000
  5 Toga Scorpio                   : 415.5/1000
  6 Toga NO TB                     : 332.0/1000
Syzygy WDL doesn't help SF at all.

Re: Performance of Syzygy and Scorpio

Posted: Wed Feb 05, 2014 10:48 am
by M ANSARI
I cannot imagine any EGTB without DTZ would be of help. The engine simply doesn't have the capacity to understand many wins as the wins require moves which if "fixed" in the algorithmic sense ... they would make the engine play poorly in the majority of other positions. Somehow I feel that there are some low lying fruit in easy ELO points gained by using a different scheme than EGTB's in chess. Of course SSD's are a huge boost and the dramatic compression that bitbases have made possible is also tremendous. But if you look at chess endgames it sometimes feels like quantum mechanics ... things at the minute level do not conform to normal physics laws. Maybe a normal CPU chess algo is not the correct way to deal with endgames and the solution lies with a daughter card (like a VGA card with many CUDA processors) that can do EGTB's on the fly and can see "no progress" and find "progress" using little chess knowledge and more Monte Carlo probing also on the fly. The abilities of the latest VGA cards is really phenomenal ... it would seem that a combination of fast memory RAM or SSD and fast CUDA processing could be the better path to take for endgames rather than just have a giant storage bank of EGTB's getting probed by a normal chess engine using an integer based CPU algo.