Extracting raw data from already-computed Syzygy tables

[Dave Gomboc]

As some of you may know, I have recently been the first author of some research regarding Chess EGTs:

IWLS 2020: http://www.cs.ucr.edu/~cshelton/papers/docs/iwls20.pdf
ACG 2021: http://www.cs.ucr.edu/~cshelton/papers/docs/acg21.pdf
ECAI 2024: http://www.cs.ucr.edu/~cshelton/papers/docs/ecai24.pdf
(More research is currently in progress.)

While I have written my own unmove generation code, for the purposes of my current research it would make sense to again take advantage of the existing data sources that have already been computed and distributed in .rtbw and .rtbz files. For the publications above, I used (an updated version of) Fathom to probe all positions of interest... one at a time. While grossly inefficient, doing so was adequate for the <= 5-figurine EGTs.

It seems that I may have an opportunity to use a larger machine that would allow me to work with larger EGTs for the next paper. However, should that occur, I won't be able to use actually use larger existing EGTs if it takes an unreasonable amount of time to extract the required information from them. So, what I'm looking for is an extraction program that can, for example, process an .rtbw file to obtain the complete set of stored positions and their associated WCDBL values without doing repeated work (e.g., decompress a chunk of compressed material just once, and extract all relevant information from that decompressed chunk on that single visit). Has someone already written such a utility?

If nobody has, then I would be happy to write and open-source such a utility, but I really can't afford to spend a lot of time concentrating on figuring out specifically how to do it: I need to keep my brain primarily concentrating on bigger-picture computing science research ideas that influence the overall course of my Ph.D. research. So, also please chime in you know how and are willing to describe easily enough how to write such a program in enough detail that I could take your description and bang out some code to do it in relatively short order.