Hash table bug

Discussion of chess software programming and technical issues.

Moderators: hgm, Rebel, chrisw

Sven
Posts: 4052
Joined: Thu May 15, 2008 9:57 pm
Location: Berlin, Germany
Full name: Sven Schüle

Re: Hash table bug

Post by Sven »

hgm wrote:I think it is a mistake to treat rights incrementally, in the key. Adding the rights keys to the (incrementally updated) board key whenever you need the hash key is faster. Doing it incrementally requires to remove the old key, in addition to adding the new key. That just doubles the work, for no benefit.
And what do you do when undoing a move?
a) Restore the board key from a stack, and recalculate the full hash key by XORing board key and rights,
b) restore the board key and the full hash key from a stack,
c) something else (what)?
User avatar
hgm
Posts: 27808
Joined: Fri Mar 10, 2006 10:06 am
Location: Amsterdam
Full name: H G Muller

Re: Hash table bug

Post by hgm »

It depends. In my latest engines I use copy-modify, so the original (incremental) key gets never changed. There is no performance penalty for the copy, as to operate on the key a copy of it has to be made in a CPU register anyway. So it is just a matter of whether you store the updated key back in the same memory location or in a different one. Copy-modify is only bad on large data structures that stay largely unmodified, where copying the unmodified part is pure overhead.

The method I now prefer is to define a struct 'Stairway', to contain all data to be shared between parent and child. I declare one in Search, and pass a pointer to it to its children in the recursive call. The hash key for the next child can be written in there by MakeMove(), derived from the hash key of its parent.

There never is a full hash key to recalculate. Each instance of Search only needs it once, for the hash probe. This calculates the full key, splits it into an index (entry number, or bucket number and slot number) and a signature. The latter reside in local variables, remaining available for the remainder of the routine. (Used when storing the search result in the TT.)

BTW, note that you can get rid of the side-to-move key completely, by defining the Zobrist key for an empty square to its value instead. This way, when you do

Code: Select all

newKey = oldKey ^ Zobrist[piece][fromSqr] ^ Zobrist[promotedPiece][toSqr] ^ Zobrist[victim][captSqr];
the Zobrist[victim][captSqr] will cause the color-flip when victim = EMPTY. In case of a capture the actual victim has to cause the color flip, which means you must imagine that your tabulated Zobrist keys are the XOR of the stm Key and some 'bare' piece key. The stm key will cancel in the from- and to-contributions for the moved piece. As the keys were random to begin with, it doesn't matter whether you generate the bare key or the key that already includes the stm key. (Caveat: this does not work in the presence of multiple captures.)
zenpawn
Posts: 349
Joined: Sat Aug 06, 2016 8:31 pm
Location: United States

Re: Hash table bug

Post by zenpawn »

At some point, I'll probably try fail soft because debugging fail hard can be very tricky. You see a crazy score, but it's not necessarily wrong, it's just a bound. I think it'd be easier to track things down when the eval score is more obviously backed up the tree.

Anyway, as mentioned earlier, this bug definitely has to do with the interplay between aspiration windows and the transposition table. Disable either one and the blunder doesn't happen and the PVs look normal.

As a middle-ground, I tried returning alpha immediately at ply 0 if the first move fails. That way, it can reset the aspiration window immediately rather than attempting the rest of the moves first. This "fixed" it, but of course, with this solution or the outright disabling of aspiration windows, I fear I'm just covering up the underlying issue, so the frustration continues. Everything else (like tablebase support) is on hold until this is resolved.
zenpawn
Posts: 349
Joined: Sat Aug 06, 2016 8:31 pm
Location: United States

Re: Hash table bug

Post by zenpawn »

Found it! This is embarrassing, but at some point in the past I noticed my engine being overoptimistic regarding repetition draws. It didn't happen with the TT disabled, so my clever idea was to just not store draw scores. It worked, but clearly it was not so clever. ;)

This blunder was due to the mate score running up against an alpha bound of, you guessed it, ~0. So, it didn't overwrite the much worse prior move in the TT and became somewhat blind to the mate.

I've removed the silly repetition draw fix, and the PVs now look as expected, i.e., similar to the ones without the TT enabled.

My question now is: Do any of you have handling around the issue (if it is one) of storing repetition draws in the TT?

Thank you all for your responses. I felt encouraged to buckle down today and find the culprit.
User avatar
cdani
Posts: 2204
Joined: Sat Jan 18, 2014 10:24 am
Location: Andorra

Re: Hash table bug

Post by cdani »

zenpawn wrote:Found it! This is embarrassing, but at some point in the past I noticed my engine being overoptimistic regarding repetition draws....
My question now is: Do any of you have handling around the issue (if it is one) of storing repetition draws in the TT?
This happened also at some point to Andscacs, thinking that it had a draw line and then losing. I did nothing special, just continue improving it, and eventually the problem vanished. So I think is not a concrete problem, just that the engine for example reduces too much or has some poor knowledge or whatever.