Ignoring castling rights and en passant state in zobrists hashing

[gflohr]

I am currently implementing zobrist hashing and I wonder whether it is really necessary to encode the en passant square and the castling state of a position. Has that been discussed before? I could not find anything about it in the search.

At first glance, it looks like ignoring these properties is clearly a bug but I am curious whether that bug could lead to wrong decisions of the engine. The falsely positive table hits caused by this omission can only occur at least four plies deeper in the search. And it implies that both sides have moved at least one of their pieces in a circle in the meantime. This may be advantageous for one side but I cannot see that such a maneuver could be optimal for both sides at the same time. To me it looks that the line leading to the false table hit can never be the optimal line which an engine is supposed to find.

And even if we get there, would it matter? If a position is evaluated again that only differs in castling rights or en passant state from a previously evaluated position, I think that the evaluation can never get better because the side to move has less options. And so that line would be refuted anyway, wouldn't it? You could also argue that having the right to castle or to capture en passant obviously did not matter in this particular position because it was not used. Okay, the castling right may still be an asset in the future but what does it change if the opponenents first move their pieces around in circles?

What did I miss here?

[R. Tomasi]

If the pawns stop square is blocked then the en passant flag might make a difference: without the flag, there may be no legal move, leading to a stalemate. I agree, that this seems to be a rather artificial case which probably never happens in a real game. But technically it makes a difference.

[mvanthoor]

What did I miss here?

The same position, without castling rights or without EP-capture, is NOT the same position. This matters when checking for move repetition. Even when it's the same position, but NOT the same color to move, it's not the same position. In practice, this is sometimes even forgotten by players.

A position in chess is the same if:
- all the same pieces are on the same squares
- all the same legal moves are possible
- the same color is to move

Take away one, and it's not the same position. Therefore castling rights, EP, and side to move are encoded in the Zobrist hash.

[gflohr]

If the pawns stop square is blocked then the en passant flag might make a difference: without the flag, there may be no legal move, leading to a stalemate. I agree, that this seems to be a rather artificial case which probably never happens in a real game. But technically it makes a difference.

Good catch!

[gflohr]

I am not sure whether I had made my point correctly. I don't doubt that two positions are indeed different if either the castling rights or en passant square had changed. I am just wondering whether an otherwise correct chess engine would ever get into a situation where it matters. Could such a sequence of moves be part of the principal variation?

[mvanthoor]

I am not sure whether I had made my point correctly. I don't doubt that two positions are indeed different if either the castling rights or en passant square had changed. I am just wondering whether an otherwise correct chess engine would ever get into a situation where it matters. Could such a sequence of moves be part of the principal variation?

Assuming has no bugs and plays only legal moves if the Zobrist key contains e-p, casting, and color, then will not play illegal moves if you omit that information from the Zobrist hash. Zobrist hash is not used in move generation. (You DO need this information in the engine itself, obviously, or you WILL miss legal moves, and either never take e-p, never castle, or worse, play twice for the same side.)

However, because the engine is unable to accurately determine if a position is actually different than the ones it saw before due to the Zobrist hash missing information, one of two things can happen:

- It plays a move leading to a position it had before and it's a real repeat, but the engine doesn't know this because the EP, Castling and possibly Color information is missing. The user interface (in case of a UCI-engine) will terminate the game as a draw. Because of this, the engine could draw positions it should have won.

- It plays a move leading to a position it THINKS is the same, trying to go for a draw, but the position actually isn't the same so the user interface doesn't terminate the game; it makes the engine play on, even though it really wouldn't if it had the choice. (And it tried to make this choice by going for the repeat.) This will make the engine lose positions it could have otherwise drawn.

So, omitting this information in the Zobrist key will not make the engine play illegal moves, but it WILL make the wrong decisions in positions where repeating either means missing the win, or not saving the draw.

You actually see this happening if an engine has a bug in either the Zobrist hashing, or in the position repetition detection. I actually encountered the latter myself due to an "off by one." Not winning games you should have won, and not drawing games that could be drawn, is a huge Elo loser.

[bob]

The danger is something like a position where black can castle. White checks black and black moves the king over. Which checks again and black moves the king back and the search calls this a repetition. It is not. Another case is two identical positions where in one case, white played a two-square pawn push, in the other it did it in two one square moves. Positions are NOT equal. The trees we search are so ridiculously large that this happens far more than you would think.

[gflohr]

Okay, thanks everybody, got it.

[j.t.]

Okay, thanks everybody, got it.

I don't think it makes a big practical difference:

Code: Select all

Score of Nalwald 14.zobrist-light vs Nalwald 14: 240 - 239 - 521  [0.500] 1000
...      Nalwald 14.zobrist-light playing White: 144 - 93 - 264  [0.551] 501
...      Nalwald 14.zobrist-light playing Black: 96 - 146 - 257  [0.450] 499
...      White vs Black: 290 - 189 - 521  [0.550] 1000
Elo difference: 0.3 +/- 14.9, LOS: 51.8 %, DrawRatio: 52.1 %

Nalwald 14.zobrist-light ignores castling and en passant states in positions for the zobrist key.

[gflohr]

I don't think it makes a big practical difference:

Code: Select all

Score of Nalwald 14.zobrist-light vs Nalwald 14: 240 - 239 - 521  [0.500] 1000
...      Nalwald 14.zobrist-light playing White: 144 - 93 - 264  [0.551] 501
...      Nalwald 14.zobrist-light playing Black: 96 - 146 - 257  [0.450] 499
...      White vs Black: 290 - 189 - 521  [0.550] 1000
Elo difference: 0.3 +/- 14.9, LOS: 51.8 %, DrawRatio: 52.1 %

Nalwald 14.zobrist-light ignores castling and en passant states in positions for the zobrist key.

That's interesting information. I will also try it out, once I have transposition tables working.

It doesn't matter though anymore for me because I have found a way to update castling en passant state at no cost.