I recently got a note from TCEC team and they were basically saying that since they are going to be running on 176 cores for the next season, they are concerned that engines be able to handle this well. They mention this testing technique suggested by the Demmolito author:
Or by randomizing the hash move. My first thought was, well, I don't have this issue, but I can make problems occur with the suggested method. One of them is this scenario: a hash move is fetched when the side to move is in check, and it would be a legal move except it is not an evasion. Furthermore that move is also a checking move. The program executes it, and in the next ply, it does not detect the illegality because of the check - it just generates evasion moves, when it fact it also has a capture of the opposing King, which would ordinarily result in illegality detection. I am pretty sure there are a few other rare conditions like this. The easiest fix would be to just go to to a 128-bit hash code, as I think Steven Edwards did for his program back once upon a time, but that does increase memory usage/reduce hash entries for a fixed amount of memory.I found a crashing bug linked to HT races. I don't know if this is the only bug causing crashes, but it sure is the most likely candidate (ie. rare crashes, whose probability of occurrence increases with nb of threads).
This might be of interest to other programmers. The debugging technique consists of simulating in single threaded search, the effect of SMP races. The important thing to note is that, even if HT read/writes are racing, the read and write operations are still atomic on 64-bit words. So the result of an HT race is not completely random, but of the form e'=(key1,data2) mixing two entries e1=(key1,data1) and e2=(key2,data2) that were racing. This can be simulated in single threaded mode (which is 100% deterministic), with as high probability of crashing as one desires, by reducing the number of bits used (ie. generate more collisions). Example using only 8-bits instead of 64-bit, and demolito crashes very easily:
if (((e->keyXorData ^ e->data) & 0xFFull) == (key & 0xFFull)) {...}