Hello Ronald de Man,
Could you describe the situation of end game tablebases when you started to work on the Syzygy ones? After the success of Gaviota tablebases which reduced a lot the size of files (due to compression I suppose), what made you think you could come up with something a bit different but as effective, and more importantly, much more "compressed"?
Thanks for your time. It's always a pleasure for me to read you even though many things are over my head.
The history of Syzygy tablebases
Moderators: hgm, Rebel, chrisw
-
- Posts: 1142
- Joined: Thu Dec 28, 2017 4:06 pm
- Location: Argentina
Re: The history of Syzygy tablebases
I would also like someone to comment on the pros and cons of the different TBs formats, we have Gaviota, Syzygy, Scopriobases, Shredderbases, Nalimov, Lomonosov, etc.. Why would you choose one over the other? What are their differences? Which ones are already considered obsolete?Isaac wrote:Hello Ronald de Man,
Could you describe the situation of end game tablebases when you started to work on the Syzygy ones? After the success of Gaviota tablebases which reduced a lot the size of files (due to compression I suppose), what made you think you could come up with something a bit different but as effective, and more importantly, much more "compressed"?
Thanks for your time. It's always a pleasure for me to read you even though many things are over my head.
Follow my tournament and some Leela gauntlets live at http://twitch.tv/ccls
-
- Posts: 5565
- Joined: Tue Feb 28, 2012 11:56 pm
Re: The history of Syzygy tablebases
I had generated 5-piece WDL+DTZ TBs for regular chess and suicide chess in 2005 with a similar compression scheme (based on "Re-Pair compression" followed by Huffman). The current generator is an almost complete rewrite which made the generator much faster and multi-threaded and which added many ideas to improve the compression, basically by not storing information that can easily be reconstructing when probing.Isaac wrote:Could you describe the situation of end game tablebases when you started to work on the Syzygy ones? After the success of Gaviota tablebases which reduced a lot the size of files (due to compression I suppose), what made you think you could come up with something a bit different but as effective, and more importantly, much more "compressed"?
-
- Posts: 2929
- Joined: Sat Jan 22, 2011 12:42 am
- Location: NL
Re: The history of Syzygy tablebases
Scorpio and (iirc) Shredder are bitbases: they store whether a position is won for the side to move or not. Typically that is enough during the search, as long as the root position is not a tablebase position. In that case you need a way to make progress, and the bitbases don't really help. However, in practice chess programs can win many endgames without tablebases.CMCanavessi wrote: I would also like someone to comment on the pros and cons of the different TBs formats, we have Gaviota, Syzygy, Scopriobases, Shredderbases, Nalimov, Lomonosov, etc.. Why would you choose one over the other? What are their differences? Which ones are already considered obsolete?
Nalimov and Gaviota store distance to mate (DTM) for the side to move (or not-won if the position is not won), but they ignore draws by the 50-move rule. Nalimov has 6 men, but comes with an obnoxious custom license. Gaviota probing code is open source under a liberal license, but it only has up to 5 men. Compression is (iirc) much better than Nalimov.
Lomonosov is DTM up to 7 pieces, but not available for free (and impractically large).
Syzygy bases store the number of moves needed to win a position under the 50-move rule (DTZ50). You win by either mating the opponent, making a winning pawn push or by converting to another won ending. The generator and probing code are open source.
For practical purposes, if you pick just one, Syzygy is probably the best pick. In a pinch, Syzygy+Gaviota might be better, since they will give you the shortest DTM in positions that are won under the 50-move rule. Syzygy alone doesn't do that. If that is better or not is a matter of taste.
Nalimov should be a no-go because of the license and Lomonosov is simply not an option.
Pure bitbases are probably the least useful these days (Syzygy and I think Gaviota actually already include bitbases for the search).
-
- Posts: 5565
- Joined: Tue Feb 28, 2012 11:56 pm
Re: The history of Syzygy tablebases
If I am not mistaken, both store W/D/L.Evert wrote:Scorpio and (iirc) Shredder are bitbases: they store whether a position is won for the side to move or not.
I think Gaviota's compression ratio is similar to Nalimov's (I see 6.5GB being mentioned where Nalimov is a bit above 7GB). I don't know how they compare in terms of decompression speed.Gaviota probing code is open source under a liberal license, but it only has up to 5 men. Compression is (iirc) much better than Nalimov.
-
- Posts: 2821
- Joined: Fri Sep 25, 2015 9:38 pm
- Location: Sortland, Norway
Re: The history of Syzygy tablebases
It would be interesting to know how Nalimov-6 performs on high-end ssds. I don't have SSDs larger than 256 GB.
Nalimov pros and cons
My observations
Pluss | Does not eat RAM during analysis
Pluss | Display depth-to-mate (DTM)
Minus | Cumbersome size
Minus | No 50-move rule
Gaviota is slightly lower in size but only limited to 5-man.
The only drawback of syzygy is no dtm information.
Nalimov pros and cons
My observations
Pluss | Does not eat RAM during analysis
Pluss | Display depth-to-mate (DTM)
Minus | Cumbersome size
Minus | No 50-move rule
Gaviota is slightly lower in size but only limited to 5-man.
The only drawback of syzygy is no dtm information.
-
- Posts: 27796
- Joined: Fri Mar 10, 2006 10:06 am
- Location: Amsterdam
- Full name: H G Muller
Re: The history of Syzygy tablebases
"Does not eat RAM during analysis"... Isn't that the same thing as saying "Fails to profit from RAM that otherwise would be idle, and just leaves that go to waste"?
-
- Posts: 558
- Joined: Sat Mar 25, 2006 8:27 pm
Re: The history of Syzygy tablebases
Not really. Suppose you have two programs: one does tons of disk thrashing if it doesn't have 8GB of RAM available, the other doesn't:hgm wrote:"Does not eat RAM during analysis"... Isn't that the same thing as saying "Fails to profit from RAM that otherwise would be idle, and just leaves that go to waste"?
Code: Select all
4GB 8GB
A 2400 2450
B 1800 2480
-
- Posts: 2821
- Joined: Fri Sep 25, 2015 9:38 pm
- Location: Sortland, Norway
Re: The history of Syzygy tablebases
Nalimov algorithm does not cache egtb in ram, this mean more ram is available for engines in analysis. Also there is no worries about 99% ram usage.
For example with syzygy-6, ram usage can go to 98 or 99% without sufficient ram installed. Windows don't perform well if ram is filled up.
For example with syzygy-6, ram usage can go to 98 or 99% without sufficient ram installed. Windows don't perform well if ram is filled up.
-
- Posts: 10948
- Joined: Wed Jul 26, 2006 10:21 pm
- Full name: Kai Laskos
Re: The history of Syzygy tablebases
Well, RAM usage is the property of an engine, like hash. Engine with larger hash is often favored. Another point: have you ran 6-men Syzygy from fast SSD on 8GB RAM machine (with say another 8GB cache on HDD) for several hours from close to endgame positions? It becomes practically unresponsive. What I did, I now have 16GB RAM and 0 cache, but my machine sometimes is in danger to simply nastily crash due to Syzygy.hgm wrote:"Does not eat RAM during analysis"... Isn't that the same thing as saying "Fails to profit from RAM that otherwise would be idle, and just leaves that go to waste"?
Sure, otherwise, Syzygy are by far the best endgame bases for all practical purposes. DTM shortened in say SF9, with Ronald's patch. The best implementation is in Texel of Peter, who made use of Syzygy + Gaviota.