How is legality of castling usually determined?

[hgm]

Generating legal moves is not an all-or-nothing matter. Eventually legality would have to be tested, or the engine would not play by the rules. The issue is how much you postpone this testing. In the most extreme case you can defer this checking to the daughter node, to the point where it generated its pseudo-legal moves and extracted the MVV-wise best capture. You can then test whether this captures a King, and if so, return a 'minus infinity' score for rejecting the move that led to it.

If you are doing staged move generation in order of decreasing victim value you could start this with a stage where you generate captures of the King, and abort the node when you find one. That saves you the time of generating other replies to illegal moves, but the test provides more overhead after a legal move; it is a lot more complex than testing whether the MVV is a King, even in the case where there are no captures.

Rather than using a general 'IsSquareAttacked' function for testing whether the opponent left its King hanging, you can make use of the fact that Kings can only be exposed by either moving them to an attacked square, or because the previously moved non-royal piece was pinned. The latter can only happen when the from-square of the preceding move is aligned with the King, which is a fast and simple test. You then only have to do the general IsSquareAttacked test when a King was moved (which can again be excluded by a very simple test, and is usually not the case).

These shortcuts only work when there was no pre-existing check; many engines thus use a special, fully-legal move generator in positions where they are in check. At the very least the moves should be screened for resolving the existing check; whether they create a new one could be handled the same way as when not in check. The type of check (contact, distant or double) determines what moves could be of potential use, and in Joker I had dedicated check-evasion generators for each case.

If you have such dedicated King-capture detection, it is usually beter not to do it in the daughter, but move it to the parent, to just before the point where you would make the move to be searched. Usually the tests can be programmed without actually making the move. E.g. a general IsSquareAttacked that you want to do to vet a generated King move can be done without actually moving the King, if you were not in check already. There is no danger that the King will illegally 'step into its own shadow', as it cannot legally cast a shadow. (This will not work in arbitrary variants, though; lame leapers can be safely blocked by a King, and hopper might have been using the King as a mount.)

You would still only spend the overhead of legality checking on moves you really want to make, and not waste any time on testing moves that will never be tried because of a beta cutoff. And it would save you the time of making the move on moves that turn out to be illegal.

As Uri mentioned, some aspects of the legal generation can actually speed up the move generation compared to generating all pseudo-legal moves. In particular, you could detect pieces that are pinned, and suppress recognizing those as your own pieces that you have to generate moves for. Usually no pieces will be pinned, and testing that is as fast as testing a single move for legality. If pieces are pinned the usual move generation will be faster, because you don't generate moves for those. The pinned pieces might have moves along the pin ray, though. In which case you should generate those with a dedicated move generator that only produces moves along the pin ray. That is usually still faster than generating all the moves of the piece.

Whether this is beneficial depends on how many sliders there are in the game that could be aligned with the King, and thus pin a single piece of the side to move that is in between those and the King. With maximally 4 sliders that could pin pieces in orthodox Chess (Q, 2R and one B) the common case is that no enemy slider is aligned for pinning, and that then ends the test. None of the moves for the non-royal pieces you later generate then has to be tested for exposing your King; they are guaranteed to be legal in this respect. The only source of illegal moves then would be King moves (or check evasions).

[jhaglund2]

654321
000000
Check your bits. If any bit is true or binary value != 0, you cannot castle.

1) Your king has been moved earlier in the game.
2) The rook that you would castle with has been moved earlier in the game.
3) There are pieces standing between your king and rook.
4) The king is in check.
5) The king moves through a square that is attacked by a piece of the opponent.
6) The king would be in check after castling.