Squash anyone?

[mar]

you may also try to train a small (or large, depending on how much distant matches you can encode) buffer to seed LZ (anything is better than nothing),
zstd already does this in "dictionary mode", see https://github.com/facebook/zstd;
say having a seed dictionary for each piece configuration

here the ratio is 3x better on small JSON files (~6k), which is not something you'd get by using a better compression algorithm, YMMV of course

btw zstd is a masterpiece and I consider it the best open source general
purpose compression algorithm. if you add IO cost to the equation,
assuming zstd with dictionary would beat LZ4 (same author) say 3x in compression ratio (a wild guess),
then it simply has to be faster than LZ4

[Cardoso]

Thanks Martin for your input.
First I'm no compression expert.
I tried 8K blocks and compression ratio improved a little bit but decompression speed came down considerably.
You see I allways decompress in my engine and stop decompressing at a key index and return the byte at that uncompressed index.
That's why I need very small blocks to keep decompression speed high, I would even go for 512bytes blocks, but compression ratio would be bad.
This is like Syzygy but I didn't mmaped at the time, but now I do.
I contacted Zstandard author and he told me the dic training code uses divsufsort.c (a prefix sorting library) and is limited to 2GB.
But what is worse is that in Zstandard the decompression code doesn't stop at a target index.
But LZ4 does stop decompression at a target index but compression ratio is poor for me (even the HC variant).
So both Zstandard and LZ4 almost work, but no cigar.
My WDL RLE scheme decompresses at almost 2 million probes per second per core.
Unfortunately for my DTM variant it is below 500k probes per second and compression ratio is way worse.

best regards,
Alvaro