TalkChess.com

rsync is nice, but very CPU-heavy, so it doesn't scale well for busy servers at all. There are variations like zsync that try to remedy some of this by not requiring an active server that checksums lots of data, but I don't think they ever caught on (and zsync requires extra storage space).

I couldn't find any checksums on your FTP server; did I miss anything?

Sesse wrote: ↑Sun Sep 09, 2018 6:27 pm I couldn't find any checksums on your FTP server; did I miss anything?

Wise FTP server not serving the file unless explicitly labeled by SELinux

Now it is fixed, ftp://ftp.chessdb.cn/pub/syzygy/7men/checksums.txt

All my files match that checksum file. Success!

(This is after I replaced the faulty KRRNvKRR.rtbz earlier today, of course.)

On my end I believe I now have a complete and correct set. On your side as you (Sesse) noticed, you had an old version of KRRNvKRR.rtbz, the give-away being the file size discrepancy compared to the lichess ftp listing.

The remaining five -- KNNPPvKR.rtbw, KQBNvKQN.rtbw, KQRPvKRB.rtbw, KRBBNvKQ.rtbz, KRNPvKQN.rtbz -- were file corruptions on my side. Even though they showed the correct file length they did not transfer correctly.

Question for Sesse. When you said that your set passes tbcheck, did you do the full check wherein the hash sum is recomputed and compared to the stored value, or did you do the quick check? The quick check only proves that the metadata block containing the sum transferred correctly. It won't catch a bad bit inside the data payload.

Demonstration...
("bad5" refers to my 5 bad files, not that the internal checksums provided by noobpwnftw are bad.)

One reason I'm an advocate of using a common external checksum algorithm such as the sha family is their wide availability. The programs sha256sum and the older md5sum are "just there" on most Linux distributions. Presumably for Windows and Mac, too. As it happens syzygy would not compile on my two home machines straight out of the git repo. It took me a while to find a compiler version and flag settings combination that work.

Thank you for your help in cross-checking. I'm in the process of one final full sweep of the files. No errors so far.

jk

A one-letter typo in the syzygy codebase: sizt_t

This branch must not be covered in the regression tests. The line is conditional compiled out by the makefile as distributed, i.e. -DUSE_ZSTD invokes the zstd library instead of lz4, which the quoted line is a part of.

jkominek wrote: ↑Mon Sep 10, 2018 7:30 am Question for Sesse. When you said that your set passes tbcheck, did you do the full check wherein the hash sum is recomputed and compared to the stored value, or did you do the quick check? The quick check only proves that the metadata block containing the sum transferred correctly. It won't catch a bad bit inside the data payload.

For every file I downloaded, I ran tbcheck (with no parameters) on it before I made it public for download. Thus, it checks that the file is self-consistent (internal checksum matches actual data). After the reference checksum file was posted here yesterday, I did a quick compare against that (ie., internal checksum matches external checksum).

So with the dust settling from the generation, where are we? Stockfish master has full support; is there a version of Fathom with 7-man support yet?

Is uTorrent/BitTortent compatible with this file set?

There's no active 7-man torrent that I know of. Of course, nothing would stop you from making one and uploading it to a tracker (but then again, it's unlikely you would find too many people interested in seeding).

noobpwnftw wrote: ↑Sun Sep 09, 2018 9:32 pm

Sesse wrote: ↑Sun Sep 09, 2018 6:27 pm I couldn't find any checksums on your FTP server; did I miss anything?

Wise FTP server not serving the file unless explicitly labeled by SELinux

Now it is fixed, ftp://ftp.chessdb.cn/pub/syzygy/7men/checksums.txt

Hi, I can't access your ftp from Thailand.
Do you interesting todo thai-chess too?

Regards,
Pond