EGTB compression

[Dann Corbit]

Latest version of LC4 using HC -9 gives a compression ratio of 2.721:1, and while it only compresses at 41 MB/s it decompresses at 4900 MB/s (1/3 the speed of memcpy). Since you only have to compress them once, the compression speed is pretty well irrelevant.

Seems likely that Ronald has already implemented this, but other systems like Gaviota TB and Nalimov TB might make advantage of it.

[syzygy]

I guess you mean LZ4, not LC4.

My generator includes LZ4 compression code, but only for compressing intermediate files during generation. This is down at the lowest compression setting (I think) because otherwise it takes too long. The point of the compressing the intermediate files is to speed up the generation process.

Some time ago I switched the default compression code from LZ4 to Facebook's ZSTD, which gave much better compression at fast settings (if I remember well).

The final TB files are compressed with a custom compression method that is unrelated to any general purpose compressor.

[Dann Corbit]

I see.
Yes, I meant LZ4.
With the HC version using -9, it gets very good compression ratio. Now, it compresses slowly using those settings but it decompresses ultra-fast.
That is why I thought it would be a good compression algorithm for tablebase files. Even though compression is slow, you only have to compress the files once. There is a very good compression ratio, and the decompression is absurdly fast (1/3 of memcpy speed).

Now, if you have a custom compression scheme that is tailored to the data, it may very well be better than a more generic routine,
But (for instance) Gaviota tablebase files use standardized compression schemes.
Nalimov files use something by Andrew Kadtech, but i wonder if it could be improved (since the files are fairly large).
I guess the compression method would not be ideal for formation of Syzygy files because the compression step is not fast.

Are you still working on DTM files? I have not heard anything on that subject for a while,