7-men Syzygy attempt

[jkominek]

Attached is a set of md5sums files for the complete set. If anyone would care to cross-verify I'd appreciate it. Back in the day of Nalimov promulgation, circa 2005-2006, md5sum signatures were considered essential to ensuring transmission correctness. I believe this is still true even though the syzygy format has the benefit of including an internal checksum. Because I started downloading immediately after generation began, I worry slightly that I have some old, invalid files, which would still nonetheless pass internal checks; noobpwnftw had to restart generation a number of times as bugs were discovered and repaired.

Private message to Josh Shriver (posted in public). If you're willing to drive from Morgantown WV to Pittsburgh I'll hand you a complete set on an external USB drive. Hook it up to Olympus Chess. Make torrents.

jk

[Sesse]

I'll be happy to verify, but could we perhaps use something slightly more modern than MD5, now that we're in 2018? =) SHA-256, perhaps?

[Sesse]

Code: Select all

pannekake:~> sudo pvs
  PV         VG        Fmt  Attr PSize   PFree 
  /dev/md127 pannekake lvm2 a--   36,38t 18,19t                                                                                                       
  /dev/md2   cache     lvm2 a--  447,14g     0 
  /dev/md3   pannekake lvm2 a--    5,46t     0 

Now for download and tbcheck…

[Robert Pope]

I'll be happy to verify, but could we perhaps use something slightly more modern than MD5, now that we're in 2018? =) SHA-256, perhaps?

Forgive my ignorance, but why would you prefer one over the other?

[Sesse]

MD5 has been broken since a long time; it's known how to (fairly easily) construct two files with the same MD5 checksum. Even though this is not the same as a second preimage attack (create a file with a given MD5 sum), this makes it fairly useless as a cryptographic hash, and it has been deprecated on the web for a long time.

In the case of tablebases, maybe it doesn't matter (it's perhaps not so likely that someone would try to construct a fake/broken tablebase on purpose). But there's still no good reason to cling onto a standard everybody is trying to get rid of as soon as possible.

[jkominek]

I like sha256. It takes twice as long as md5 to generate and verify, though. Or, half the time of sha512, if you want to look at it that way.

[Sesse]

Well, twice as much CPU. If you have a few cores, you're likely to become I/O bound either way, though.

Thanks for the file; when I'm done downloading and have run tbcheck on everything, I'll verify on my side.

[syzygy]

Comparing file sizes is sufficient to ensure that your files are up to date and do not include any of the files that were found to be incorrect due to problems with the generator.

The internal checksum is sufficient to ensure that your files were not corrupted during transmission or while being stored on disc.

None of this is cryptographically secure, but that is fine as long as you are not downloading files from an unknown source. (But what are the chances that some evil hacker chooses to mess with multigigabyte tablebase files? There are easier targets than that.)

[Sesse]

Yes; it's more about eradicating MD5 from the face of the planet than genuine worry about malicious actors.

[Sesse]

tablebase.sesse.net now has the full 7-man Syzygy set. All files have been checked with tbcheck.

I'm now running sha256sum, which will take 7–8 hours or so; it's I/O limited, so the box will be quite sluggish while it's going on. After that, there's some setup and reboots to get the SSD caching back on track, but after that, it's all good.