ep and castle rights hashing

[xmas79]

Hi all,
recent post about ep square on polyglot,pointed me to having that ep "bug" in my engine,as I blindly set ep square on every double push.fixing this was very easy,but now I think I should do the same thing on castle rights. when in check, a king cannot castle,so search should not distinguish between two positions where one king is in check regardless of the castle rights for that king. probabily is not worth the effort? Should I do this?

best regards,
Natale.

The "is in check" property is not the key point here. The real point is that you can have two positions with identical piece locations where in one case a castling right is available (so king and rook have not moved yet) and in the other case that castling right is not available (so either the king or the rook has already moved but then returned to its original square). Whether the king is in check in both cases, or is not in check in both cases, does not matter at all.

As to the hashing aspects of this situation, everything has already been said, i.e. different castling rights require different hash keys and also require to recognize the two positions as different w.r.t. repetition detection, where the latter is based on the FIDE rules. So even if, in the "in check" case, the available legal moves are the same in that position they are not necessarily the same in positions later in the game. En passant is different here since it is a short-term property of the position that is valid for one ply only, while each of the four castling rights remains valid until a move is played that destroys it.

Sven

From a hashing POV I already have different keys for different castle rights, so I'm doing it right. With respect to FIDE rules, instead, it seems to me that 9.2 doesn't speak very clear:

Positions as in (a) and (b) are considered the same, if the same player has the move, pieces of the same kind and colour occupy the same squares, and the possible moves of all the pieces of both players are the same.

That part says when two positions are identical. And in the case of the situation where a king is in check a castle move is temporary forbidden, and the legal moves available to the color being in check are the same regardless of its castling rights. Notice the "its" castling rights, as to verify if two positions are the same we should generate all the possible moves from both colors and exactly match them in the two positions, so if the "checking" side can or not can castle makes difference here. And this is obviously not true for the "checked" side, where its moves are always the same.

Positions are not the same if a pawn that could have been captured en passant can no longer be captured in this manner. When a king or a rook is forced to move, it will lose its castling rights, if any, only after it is moved

The first part of this sentence seems to me a clarification, even if it seems pretty obvious that if I can no more take a pawn EP in a position when I could before, I don't have exactly the *same* moves available, so the two positions are not identical.
The second part, instead, clarifies that the castling rights are lost only after moving rooks/kings, but it has nothing to do with respect to identifying if two positions are equal.

So the question was: should we remove the castling rights from hash signature when a king is in check and restore them accordingly to the escape move in order to be "FIDE compliant"? Or this situation is going to happen so few times that I don't bother? As someone else pointed out, considering two positions being the same (with the same hash) will allow "perfect" repetition verification, but search will be messed up, as an hash hit in a such position *will* produce unexpected results because it will not know anything about the castling rights...
The answer is: always keep castling rights until they are effectively lost by a king/rook move.

The EP story is the same: always set EP square after a double push so you will win a "FIDE compliant" award by messing up your search, as it will lead to worse search results.


PS
Obviously, when not is check all this stuff is garbage, as castling rights will have impact on position recognition...

[tpetzke]

The EP story is the same: always set EP square after a double push so you will win a "FIDE compliant" award by messing up your search, as it will lead to worse search results.

Don't hope for a FIDE compliant award. You might get one from Steven Edwards for complying to to FEN standard.

Thomas...

[Sven]

Hi all,
recent post about ep square on polyglot,pointed me to having that ep "bug" in my engine,as I blindly set ep square on every double push.fixing this was very easy,but now I think I should do the same thing on castle rights. when in check, a king cannot castle,so search should not distinguish between two positions where one king is in check regardless of the castle rights for that king. probabily is not worth the effort? Should I do this?

best regards,
Natale.

The "is in check" property is not the key point here. The real point is that you can have two positions with identical piece locations where in one case a castling right is available (so king and rook have not moved yet) and in the other case that castling right is not available (so either the king or the rook has already moved but then returned to its original square). Whether the king is in check in both cases, or is not in check in both cases, does not matter at all.

As to the hashing aspects of this situation, everything has already been said, i.e. different castling rights require different hash keys and also require to recognize the two positions as different w.r.t. repetition detection, where the latter is based on the FIDE rules. So even if, in the "in check" case, the available legal moves are the same in that position they are not necessarily the same in positions later in the game. En passant is different here since it is a short-term property of the position that is valid for one ply only, while each of the four castling rights remains valid until a move is played that destroys it.

Sven

From a hashing POV I already have different keys for different castle rights, so I'm doing it right. With respect to FIDE rules, instead, it seems to me that 9.2 doesn't speak very clear:

Positions as in (a) and (b) are considered the same, if the same player has the move, pieces of the same kind and colour occupy the same squares, and the possible moves of all the pieces of both players are the same.

That part says when two positions are identical. And in the case of the situation where a king is in check a castle move is temporary forbidden, and the legal moves available to the color being in check are the same regardless of its castling rights. Notice the "its" castling rights, as to verify if two positions are the same we should generate all the possible moves from both colors and exactly match them in the two positions, so if the "checking" side can or not can castle makes difference here. And this is obviously not true for the "checked" side, where its moves are always the same.

Positions are not the same if a pawn that could have been captured en passant can no longer be captured in this manner. When a king or a rook is forced to move, it will lose its castling rights, if any, only after it is moved

The first part of this sentence seems to me a clarification, even if it seems pretty obvious that if I can no more take a pawn EP in a position when I could before, I don't have exactly the *same* moves available, so the two positions are not identical.
The second part, instead, clarifies that the castling rights are lost only after moving rooks/kings, but it has nothing to do with respect to identifying if two positions are equal.

So the question was: should we remove the castling rights from hash signature when a king is in check and restore them accordingly to the escape move in order to be "FIDE compliant"? Or this situation is going to happen so few times that I don't bother? As someone else pointed out, considering two positions being the same (with the same hash) will allow "perfect" repetition verification, but search will be messed up, as an hash hit in a such position *will* produce unexpected results because it will not know anything about the castling rights...
The answer is: always keep castling rights until they are effectively lost by a king/rook move.

The EP story is the same: always set EP square after a double push so you will win a "FIDE compliant" award by messing up your search, as it will lead to worse search results.


PS
Obviously, when not is check all this stuff is garbage, as castling rights will have impact on position recognition...

For me there is no doubt that the FIDE rules are clear in stating that a difference in castling rights of two positions implies that these positions are not identical. I see no other way for the sentence "When a king or a rook is forced to move, it will lose its castling rights, if any, only after it is moved." to make any sense since otherwise it would be completely redundant. Why else should they mention that a king in check (which is in fact the only case that can be meant by the rule) still keeps its castling rights and loses them only after moving? No other aspect of the rules is affected by that, apart from the question whether two positions are identical.

Also for me there is no doubt that a difference in castling rights of two otherwise identical positions, even if the king is in check, can lead to different search results even though in the root position both have the same set of legal moves.

Therefore I don't see why you insist here.

Sven

[hgm]

Indeed, I fully agree. This is also how WinBoard treats it. And it actually does occur, this is not a purely hypothetical quirk. Often a perpetual check starts on a King that still has castling rights. Some engines do claim one ply too early in that case, and are forfeited for the false claim.

I guess the remark in the FIDE rules (which, though not redundant at first glance seems a bit obvious) was added because from a game-theoretical point of view a position where a piece had not moved before, but is forced to move because it is the only way to evade check, would be completely equivalent to one where it had moved before (as far as future moves are concerned).

[syzygy]

So the question was: should we remove the castling rights from hash signature when a king is in check and restore them accordingly to the escape move in order to be "FIDE compliant"?

FIDE rules say exactly nothing about what you should put into your hash signature. Look at the rules, they are silent on hash signatures.

However, if you use hash signatures to detect 3-fold repetition, then you'd better including castling rights in the hash signature.

And if you do that, then removing castling rights when a king is in check is wrong, as has already been explained in detail. You probably have to be more careful when reading the above comments and the FIDE rules.

In particular, the FIDE rule you cite mentions "possible moves of all the pieces of both players" which implies that you should not only look at the legal moves of the side to move (the other side not having any legal moves, as it is not its turn to move), but at all legal continuations. The next paragraph then makes one clarification/exception: castling rights are only lost after the king or rook is moved. The only possible reason for mentioning this, is that it qualifies the preceding paragraph.

edit: I agree 100% with Sven on this.

[hgm]

It is often not desirable to use the same key for repetition detection and TT hashing. In Shokidoki I use only the 32-bit signature part of the hash key, and I make sure Zobrist keys for aspects of the position that I do not want to affect repetition detection have the part that contributes to the signature zeroed.

In particular these are the keys for pieces in hand. So the signature is only dependent on board position. In the repeat stack I store that signature together with the (incrementally updated) material evaluation of the position (including in-hand pieces).

This way I detect true repeats as well as pseudo-repeats (same board, different hands). As extra pieces in hand is another case of certain upward compatibility (like extra castling rights in Chess), a pseudo-repeat of a position where a piece or Pawn is transfered from one hand to the other should be an absolute no-no for the side losing the piece, and receives a -INF score for that side. I check for this by the eval difference being exactly two Pawns in hand, or being larger or equal to the difference of the in-hand value of the strongest and the weakest piece (Rook and Lance). Lower differences point to swapping pieces between hands, and no conclusion can be drawn from this. (The weaker piece might be just what you need to checkmate, so it might be good play to sac the stronger piece for it.)

[syzygy]

I guess the remark in the FIDE rules (which, though not redundant at first glance seems a bit obvious) was added because from a game-theoretical point of view a position where a piece had not moved before, but is forced to move because it is the only way to evade check, would be completely equivalent to one where it had moved before (as far as future moves are concerned).

A somewhat similar point arises if there is a "pawn attacking a square crossed by an opponent’s pawn which has advanced two squares in one move from its original square" (Article 3.7(d)) and "on the move following this advance" the 'en passant' capture is illegal.

That such a position is the "same" as the position without e.p. rights follows from a strict application of Article 9.2 "the possible moves of all the pieces of both players are the same". This is then confirmed by "Positions are not the same if a pawn that could have been captured en passant can no longer be captured in this manner." (One could however argue that "could have been captured" also covers the case that the capture, which is legal according to Article 3.7(d), is illegal due to "Leaving one's own king under attack, exposing one's own king to attack" (Article 1.2). I admit that Article 3.7(d) states "is only legal" and not "is legal".)

The same strict application of "the possible moves of all the pieces of both players are the same" to castling rights is incorrect due to the remark that castling rights are only lost after the king or rook move.

(It is interesting that Article 1.2 forbids 'capturing' the opponent's king, which should not be possible in the first place. I suppose this is directly related to Article 7.)

[rjgibert]

Why not this?
[D]8/8/8/8/8/7k/4q2P/6KR w K -

[xmas79]

Hi all,
recent post about ep square on polyglot,pointed me to having that ep "bug" in my engine,as I blindly set ep square on every double push.fixing this was very easy,but now I think I should do the same thing on castle rights. when in check, a king cannot castle,so search should not distinguish between two positions where one king is in check regardless of the castle rights for that king. probabily is not worth the effort? Should I do this?

best regards,
Natale.

The "is in check" property is not the key point here. The real point is that you can have two positions with identical piece locations where in one case a castling right is available (so king and rook have not moved yet) and in the other case that castling right is not available (so either the king or the rook has already moved but then returned to its original square). Whether the king is in check in both cases, or is not in check in both cases, does not matter at all.

As to the hashing aspects of this situation, everything has already been said, i.e. different castling rights require different hash keys and also require to recognize the two positions as different w.r.t. repetition detection, where the latter is based on the FIDE rules. So even if, in the "in check" case, the available legal moves are the same in that position they are not necessarily the same in positions later in the game. En passant is different here since it is a short-term property of the position that is valid for one ply only, while each of the four castling rights remains valid until a move is played that destroys it.

Sven

From a hashing POV I already have different keys for different castle rights, so I'm doing it right. With respect to FIDE rules, instead, it seems to me that 9.2 doesn't speak very clear:

Positions as in (a) and (b) are considered the same, if the same player has the move, pieces of the same kind and colour occupy the same squares, and the possible moves of all the pieces of both players are the same.

That part says when two positions are identical. And in the case of the situation where a king is in check a castle move is temporary forbidden, and the legal moves available to the color being in check are the same regardless of its castling rights. Notice the "its" castling rights, as to verify if two positions are the same we should generate all the possible moves from both colors and exactly match them in the two positions, so if the "checking" side can or not can castle makes difference here. And this is obviously not true for the "checked" side, where its moves are always the same.

Positions are not the same if a pawn that could have been captured en passant can no longer be captured in this manner. When a king or a rook is forced to move, it will lose its castling rights, if any, only after it is moved

The first part of this sentence seems to me a clarification, even if it seems pretty obvious that if I can no more take a pawn EP in a position when I could before, I don't have exactly the *same* moves available, so the two positions are not identical.
The second part, instead, clarifies that the castling rights are lost only after moving rooks/kings, but it has nothing to do with respect to identifying if two positions are equal.

So the question was: should we remove the castling rights from hash signature when a king is in check and restore them accordingly to the escape move in order to be "FIDE compliant"? Or this situation is going to happen so few times that I don't bother? As someone else pointed out, considering two positions being the same (with the same hash) will allow "perfect" repetition verification, but search will be messed up, as an hash hit in a such position *will* produce unexpected results because it will not know anything about the castling rights...
The answer is: always keep castling rights until they are effectively lost by a king/rook move.

The EP story is the same: always set EP square after a double push so you will win a "FIDE compliant" award by messing up your search, as it will lead to worse search results.


PS
Obviously, when not is check all this stuff is garbage, as castling rights will have impact on position recognition...

For me there is no doubt that the FIDE rules are clear in stating that a difference in castling rights of two positions implies that these positions are not identical. I see no other way for the sentence "When a king or a rook is forced to move, it will lose its castling rights, if any, only after it is moved." to make any sense since otherwise it would be completely redundant. Why else should they mention that a king in check (which is in fact the only case that can be meant by the rule) still keeps its castling rights and loses them only after moving? No other aspect of the rules is affected by that, apart from the question whether two positions are identical.

Also for me there is no doubt that a difference in castling rights of two otherwise identical positions, even if the king is in check, can lead to different search results even though in the root position both have the same set of legal moves.

Therefore I don't see why you insist here.

Sven

I was not insisting, as it is pointless from an engine point of view because search will suffer a lot from this artifact. But here I saw only "in my opinion" and so on, no-one that said "the rule says...." or "the rule is clear", as the rule being written in this way can lead to different interpretations. FIDE could simply add two words in the middle of the rule:

Case 1:

Positions as in (a) and (b) are considered the same, if the same player has the move, pieces of the same kind and colour occupy the same squares, and the possible legal moves of all the pieces of both players are the same.

The rest of the rules can be removed as it becomes clearly redundant.


Case 2:

Positions as in (a) and (b) are considered the same, if the same player has the move, pieces of the same kind and colour occupy the same squares, and the possible moves of all the pieces of both players are the same. Positions are not the same if a pawn that could have been captured en passant can no longer be captured in this manner, or if the two positions have different castling rights.

I would like to have a clarification from some FIDE guy that tells us the true story...

But now, since we are here I insist on this, and this is interesting: about 9 months ago
http://www.open-aurec.com/wbforum/viewt ... =4&t=52677

Positions are not the same if a pawn that could have been captured en passant can no longer in this manner be captured or if the right to castle has been changed temporarily or permanently.

This seems to be the same FIDE rule dated back to 2005. That was pretty clear:
1) if in position A I can castle and in position B I cannot castle they are different. This is the case where I moved a king/rook during the moves that leads to the same position
2) if in position A I cannot castle and in position B I cannot castle they are the same. This can either be:
2a) I had already lost castling rights because I moved king/rook, so I lost PERMANENT castling rights
2b) I'm in check in the first position of the repetition, so I've lost TEMPORARILY the rights to castle:
2b1) I'm in check in the second position of the repetition, so I've still lost TEMPORARILY the rights to castle, but I could castle in future if I don't move king/rook
2b2) I'm in check in the second position of the repetition, but I moved the king before reaching this second position and lost the PERMANENT rights.

Case 1) and 2a) are pretty classical so no clarification is required: you explicitly moved your king or rook, so you cannot castle.
Case 2b1) and 2b2) are the case i'm questioning here: they are considered the same from the old rule, what about the new rule?

Suppose this position after 1. ... Bb4+
[d]4k3/8/8/8/1b6/8/8/4K2R w K - 1 2
In this position white cannot castle as it lost temporarily the rights to castle.
After 2. Ke2 Bf8 3. Ke1 Bb4+ the position is here again, with permanent castling rights lost. Everyone here says that these two positions are the DIFFERENT. Old rule says they are different. New rule doesn't say anything about castling rights, so they could be the same, as the moves from BOTH sides ARE the same... Previous rule was clear, now it's not IMO.

So, following the new rule, IMO having different castling rights when NOT IN check makes two positions with identical pieces placement (and no EP capture possible in both) different. IMO having different castling rights when IN check makes two positions with identical pieces placement (and no EP capture possible in both) the same position.


Guessing why the changed that rule removing the castling rights from "position matching"?

Natale.

[xmas79]

I guess the remark in the FIDE rules (which, though not redundant at first glance seems a bit obvious) was added because from a game-theoretical point of view a position where a piece had not moved before, but is forced to move because it is the only way to evade check, would be completely equivalent to one where it had moved before (as far as future moves are concerned).

A somewhat similar point arises if there is a "pawn attacking a square crossed by an opponent’s pawn which has advanced two squares in one move from its original square" (Article 3.7(d)) and "on the move following this advance" the 'en passant' capture is illegal.

That such a position is the "same" as the position without e.p. rights follows from a strict application of Article 9.2 "the possible moves of all the pieces of both players are the same". This is then confirmed by "Positions are not the same if a pawn that could have been captured en passant can no longer be captured in this manner." (One could however argue that "could have been captured" also covers the case that the capture, which is legal according to Article 3.7(d), is illegal due to "Leaving one's own king under attack, exposing one's own king to attack" (Article 1.2). I admit that Article 3.7(d) states "is only legal" and not "is legal".)

The same strict application of "the possible moves of all the pieces of both players are the same" to castling rights is incorrect due to the remark that castling rights are only lost after the king or rook move.

(It is interesting that Article 1.2 forbids 'capturing' the opponent's king, which should not be possible in the first place. I suppose this is directly related to Article 7.)

Hi,
How you would interpret this position?

[d]3r4/4p3/8/3P3k/8/3K4/8/8 b - - 0 1
After 1. ... e5 2. Ke3 Kg5 3. Kd3 Kh5 we have the same position or not? It is the same, even if everyone here that has a pseudo-legal move generator I guess will stuff ep square into zobrist hash key, having a hash bug in the engine.... From a repetition point of view, stuffing the ep square is wrong, but this has nothing to do with the performance of the engine, aka if is it worth doing all this stuff...

Best regards,
Natale.