Dann Corbit wrote:Some (such as LZ4 HC) compress very slowly but get a very good compression ratio and decompress with blazing speed.
That sort of thing is IDEAL for chess endgame database systems.
But many of these compression algorithms only work well with very large block sizes, which is not ideal for endgame databases that need to be probed during the search.
Another thing is that when storing a chess endgame database, it is possible to leave out a lot of information that can be quickly recomputed. A dedicated compression algorithm can make better use of this fact than a general purpose compressor.
could you please provide the final compression ratio for the Syzygy TBs?
Just to have an idea of the compression efficiency of your algorithm.
Also could you please elaborate on how your compression algorithm can make better use of leaving out information that can be quickly recomputed?
My WDL checkers database is compressed with a RLE scheme.
In my DTM checkers database I use another compressor.
Like you I prefer to decompress until I find the key index every time I probe.
I use a very fast and pure LZ decompressor (no Huffman), with 4K block sizes for fast decompression.
But I'm having poor compression ratio: 12% of the uncompressed size.
I set as "Don't Cares" positions with captures for the side to move and for the side not to move.
If your compression ratio is better then I'm thinking in doing something like you are doing. At this precise moment I'm compressing my 8man DTM checkers database. The uncompressed size is about 460GB, I'm expecting about 60GB for the compressed size, now this is a bit poor as a compression ratio. So could you please explain how you compression algorithm takes advantage of information easily recomputed?