7-men Syzygy attempt

Discussion of chess software programming and technical issues.

Moderators: hgm, Rebel, chrisw

noobpwnftw
Posts: 560
Joined: Sun Nov 08, 2015 11:10 pm

Re: 7-men Syzygy attempt

Post by noobpwnftw »

Sharaf_DG wrote: Sat Jun 02, 2018 8:59 am Just found out that two tables (KQvK.rtbz & KRvK.rtbz) in ftp://112.73.74.24/pub/syzygy/3-6men failed the md5 check of http://kirill-kryukov.com/chess/tableba ... -DTZ-3-4-5. Is the tables correct or is Kiril md 5 incorrect...

D:\syzygy\3-6men>fsum -c Syzygybases-DTZ-3-4-5.md5
SlavaSoft Optimizing Checksum Utility - fsum 2.52.00337
Implemented using SlavaSoft QuickHash Library <www.slavasoft.com>
Copyright (C) SlavaSoft Inc. 1999-2007. All rights reserved.
FAILED MD5 KQvK.rtbz
FAILED MD5 KRvK.rtbz
2 checksums failed
https://github.com/syzygy1/tb/commit/57 ... 4cb6e4fabb
I believe mines are correct.
Sesse
Posts: 300
Joined: Mon Apr 30, 2018 11:51 pm

Re: 7-men Syzygy attempt

Post by Sesse »

Your URL to the MD5 checksums doesn't lead anywhere, so it's hard to say. But these are the checksums I've got (and they're verified with tbcheck):

04b8d08bde6d4e004040d05f0aa6d1c9 /srv/tablebase6/www/syzygy/3-4-5/KQvK.rtbz
715f23e13f4a1dbf2eeaf0f1972f3e8b /srv/tablebase6/www/syzygy/3-4-5/KRvK.rtbz

(That's a curious MD5 pattern on the former, by the way!)
noobpwnftw
Posts: 560
Joined: Sun Nov 08, 2015 11:10 pm

Re: 7-men Syzygy attempt

Post by noobpwnftw »

Mines are different, I got:
md5sum
ac866466e16eb19a4f8c796f8e1abd2b KQvK.rtbz
9cb0795fa43904a3e91bf749971964b8 KRvK.rtbz

tbcheck
KQvK.rtbz: c51fa4b08478f04272ec668ce70d6ca7
KRvK.rtbz: 1df564d7e7056f15d9a1d776e8fe5884

DTZ tables, please refer to https://github.com/syzygy1/tb/commit/e5 ... 01a2b83650

Both are correct but checksums are different due to https://github.com/syzygy1/tb/commit/f8 ... 29194271d5.
Sesse
Posts: 300
Joined: Mon Apr 30, 2018 11:51 pm

Re: 7-men Syzygy attempt

Post by Sesse »

There's an issue with one of the latest tables and my patched Stockfish; the hash key becomes 0xf9247fff, which ends with 15 consecutive binary ones. The 12-bit hash table wants to make sure there's nothing in the last slot, so it bombs out…

Increasing this 4K table to 64K entries sounds really bad. I wonder if I should just XOR the key with some value before insert/lookup? Does anyone know if the 4K size should be increased for 7-man despite such a fix?
User avatar
Nordlandia
Posts: 2821
Joined: Fri Sep 25, 2015 9:38 pm
Location: Sortland, Norway

Re: 7-men Syzygy attempt

Post by Nordlandia »

Question for Seese: do you use some of the already generated 7-men sets for live website or wait until complete set is finished?
syzygy
Posts: 5563
Joined: Tue Feb 28, 2012 11:56 pm

Re: 7-men Syzygy attempt

Post by syzygy »

Sesse wrote: Sun Jun 03, 2018 4:45 pm There's an issue with one of the latest tables and my patched Stockfish; the hash key becomes 0xf9247fff, which ends with 15 consecutive binary ones. The 12-bit hash table wants to make sure there's nothing in the last slot, so it bombs out…

Increasing this 4K table to 64K entries sounds really bad. I wonder if I should just XOR the key with some value before insert/lookup? Does anyone know if the 4K size should be increased for 7-man despite such a fix?
I'm not sure why SF needs the last entry to be empty.

I do prefer the 1-dimensional array over my original bucket-based approach, so I implemented something similar in Cfish. But in Cfish I simply wrap the index:

Code: Select all

static void add_to_hash(void *ptr, Key key)
{
  int idx;

  idx = key >> (64 - TB_HASHBITS);
  while (tbHash[idx].ptr)
    idx = (idx + 1) & ((1 << TB_HASHBITS) - 1);

  tbHash[idx].key = key;
  tbHash[idx].ptr = ptr;
}

Code: Select all

  int hashIdx = key >> (64 - TB_HASHBITS);
  while (tbHash[hashIdx].key && tbHash[hashIdx].key != key)
    hashIdx = (hashIdx + 1) & ((1 << TB_HASHBITS) - 1);
  if (!tbHash[hashIdx].ptr) {
    *success = 0;
    return 0;
  }
According to Wikipedia, for good performance it is important that the load factor remains below 0.7. Each table needs two entries (except for the symmetrical 2v2 and 3v3 ones), which gives about 3000 entries for all 7-piece tables. So 4096 should be fine at the moment but may become a bit too small when the 7-piece tables near completion. This can be checked by calculating the average number of tries when inserting the keys.

(Instead of going to 8192 entries one could also use the trick that is now used to index the TT and increase the size by much less. One can then pick a size for which add_to_hash() happens to never exceed the last entry, so you don't even need to do any wrapping.)
syzygy
Posts: 5563
Joined: Tue Feb 28, 2012 11:56 pm

Re: 7-men Syzygy attempt

Post by syzygy »

Of course the reason why SF wants to keep the last entry empty is exactly to avoid having to wrap the index.
syzygy wrote: Sun Jun 03, 2018 5:36 pm One can then pick a size for which add_to_hash() happens to never exceed the last entry, so you don't even need to do any wrapping.
It would be sufficient to make sure that during insertion of all 7-men keys the index does not exceed the size of the table.

If the user has the complete 7-men set, any probe of the hash table will be successful, so there is no need to have an empty entry at the end of the table. This assumes that the TBs are probed only for positions with 7 men or less.

If the user does not have the complete 7-men set, not all probes will be successful, but there is still no chance of a boundary overflow since the probe will stop at the empty entry into which add_to_hash() would have inserted the key. Since we know that add_to_hash() does not overflow, everything is fine.

To play it safe, one could make the hash table array 1 entry larger. Then probes of arbitrary keys are guaranteed to stay within the allocated array. This could also solve your current problem:

Code: Select all

Entry hashTable[Size + N];

...

        // Ensure last element is empty to avoid overflow when looking up
        for ( ; entry - hashTable < Size + N - 1; ++entry)
            if (std::get<KEY>(*entry) == key || !std::get<WDL>(*entry)) {
                *entry = std::make_tuple(key, wdl, dtz);
                return;
            }
Try with N=1. If that stops working, try N = 2, etc.

So it is in fact always easy to avoid the need for the wrapping operation.
Sesse
Posts: 300
Joined: Mon Apr 30, 2018 11:51 pm

Re: 7-men Syzygy attempt

Post by Sesse »

Nordlandia wrote: Sun Jun 03, 2018 5:14 pm Question for Seese: do you use some of the already generated 7-men sets for live website or wait until complete set is finished?
I use what is available. At some point, I will run out of disk space, though.
Sesse
Posts: 300
Joined: Mon Apr 30, 2018 11:51 pm

Re: 7-men Syzygy attempt

Post by Sesse »

syzygy wrote: Sun Jun 03, 2018 5:36 pm According to Wikipedia, for good performance it is important that the load factor remains below 0.7. Each table needs two entries (except for the symmetrical 2v2 and 3v3 ones), which gives about 3000 entries for all 7-piece tables. So 4096 should be fine at the moment but may become a bit too small when the 7-piece tables near completion. This can be checked by calculating the average number of tries when inserting the keys.
Mm, OK. The modern fix for this is Robin Hood hashing, but I don't see any code for this in Stockfish. And I don't know how often this hash table is used anyway?

For the immediate workaround, I XOR-ed every key by the value 3—brute-force checking show that this makes no key end in more than eight one-bits, although of course overflow will fudge these results. At least it stopped crashing :-)
Sesse
Posts: 300
Joined: Mon Apr 30, 2018 11:51 pm

Re: 7-men Syzygy attempt

Post by Sesse »

syzygy wrote: Sun Jun 03, 2018 6:01 pm To play it safe, one could make the hash table array 1 entry larger. Then probes of arbitrary keys are guaranteed to stay within the allocated array. This could also solve your current problem:
I guess this is the most elegant solution; one can even do it runtime, adding more elements as needed.