XBoard and chess variants

[Evert]

Well, the spec (source: http://www.gnu.org/software/xboard/engine-intf.html) says this:

setup FEN
setup (PIECETOCHAR) FEN
setup (PIECETOCHAR) WxH+S_PARENTVARIANT FEN

The engine can optionally send a setup command to the GUI in reply to the variant command. In the simplest form this sends the FEN of the initial position. This can be used to implement engines for non-standard variants that only differ from standard variants through the initial position. (E.g. many of the 'wild' boards you can play on an ICS.) Whether the GUI should obey or ignore this command depends on the situation. Normally it would ignore it in variants where it knows the standard initial position and legality testing is on, or when the user specified an initial position. In other cases it will use the FEN sent by the first engine for setting up the initial position, as if it was an externally supplied position. Such a position will always be sent to a second engine that might be involved, and any setup commands received from the latter will always be ignored. (This to allow for shuffle games, where the two engines might pick different setups.) When no initial position is known, such as for 'catch-all' variants like fairy, or whenever the board width is overruled to a non-standard value, the FEN will be used as default initial position even when legality testing is on.

Optionally the meaning of the piece ID letters in the FEN can be defined between parentheses; this will be interpreted as if it was the value of a -pieceToCharTable command-line option, mapping letters to GUI piece types. Also optionally behind that, the setup command can specify board width W, board height H and holdings size S, as well as a 'parent variant'. This is typically done in response to a variant command with a non-standard name, about which the GUI is not supposed to know anything at all. The engine can then specify board size, participating pieces, initial setup, and other rule details (inherited from the parent variant), saving the user the trouble to configure the GUI for this non-standard variant.

I would assume that not specifying a pieceToChar defaults to "PNBRQFEACWMOHIJGDVLSUKpnbrqfeacwmohijgdvlsuk", while not specifying a variant would default to 8x8+0, with any (known) fairy piece possible, a single king, normal pawns and promotion on the 8th rank (basically, chess with fairy pieces). Of course the dangerous thing about assumptions is that they could turn out to be horribly incorrect.

So which fields are actually optional here? Or should I just always send the full thing?

[hgm]

It depends on the situation, and the different formats reflect the evolution of the power of this command. The engine has to send what it wants to change. The default is the current setting.

So the FEN alone would be fine if you want to keep pieceToChar, variant and board size. That quickly turned out not to be good enough in most cases, as even in Los Alamos Chess it would lead to offering you the Bishop as promotion choice. (But the engine would refuse it, so it was just a cosmetic flaw. XBoard would offer you King as promotion choice anyway, with legality testing off, presumably because at one time you had to play Suicide this way.)

In mini-Shogi leaving L and N in the pieceToChar is fatal, and lead to demotion of captured Gold and Silver to Pawn. So it was not possible to have an engine reply to "variant 5x5+5_shogi" (generated from the New Variant menu with the aid of the size overrides) with a plain "setup FEN" to play mini-Shogi; you would still have to pre-configure the pieceToChar. So the latter was optionally added to the "setup" command. But if you are happy with the current pieceToChar, such as in most of the ICS 'wild' boards, the simple format is good enough.

Later the idea of engine-defined variant names came up to obviate the clumsy use of "Variant xxx selects" type options, which would always have to be set on both engines. But since the GUI in such a case doesn't have the faintest idea of what the parent variant is, it has to be encoded in the "setup" command. Otherwise you would stay in 'VariantUnknown', which is fatal. I guess I could have chosen another default that was valid, but that would probably just make the whole mechanism more error prone.

[Evert]

It depends on the situation, and the different formats reflect the evolution of the power of this command. The engine has to send what it wants to change. The default is the current setting.

Ok, that would be very good to document explicitly. Especially since it isn't necessarily obvious (to the engine, or actually to me) what the GUI would consider to be the "current" setting in most cases.

Later the idea of engine-defined variant names came up to obviate the clumsy use of "Variant xxx selects" type options, which would always have to be set on both engines. But since the GUI in such a case doesn't have the faintest idea of what the parent variant is, it has to be encoded in the "setup" command. Otherwise you would stay in 'VariantUnknown', which is fatal. I guess I could have chosen another default that was valid, but that would probably just make the whole mechanism more error prone.

Well, part of that can be cured by documenting what the default is. For instance, I have no idea what "VariantUnknown" means precisely, why the GUI would be in that state and what exactly that implies.

Much of this is probably fairly obvious if you know how XBoard's internals work, but I don't.

[hgm]

It should be obvious to the engine what the current variant is, because the GUI just told it that in the 'variant' command. The pieceToChar would be the default for that variant. (Unless the user overruled that, but overruling a pieceToCharTable in a way the engine would not understand would wreck things in all cases.)

Indeed the docs could be somewhat more explicit on defaults. VariantUnknown would be what caused a fatal error in XBoard when a non-defined wilde type was secified (e.g. -variant wild70). Everything it doesn't know now is recognized as VariantUnkown, but the engine gets a chance to change that into something known through a setup command before it gets fatal.

[Evert]

It should be obvious to the engine what the current variant is, because the GUI just told it that in the 'variant' command. The pieceToChar would be the default for that variant. (Unless the user overruled that, but overruling a pieceToCharTable in a way the engine would not understand would wreck things in all cases.)

Sure, I know that the current variant is MyAwesomeNewChessVariant357, because I defined that in my options list, XBoard placed it on a button and the user selected that, which prompted XBoard to send me "variant MyAwesomeNewChessVariant357". It's now up to me to tell XBoard what that means.
If I understand correctly, that would require me to keep track of whatever the previously selected variant was, as well as understand what that means in terms of pieceToChar - which is tricky because these aren't actually documented anywhere as far as I can tell (this would be a good thing to have in the documentation, along with the startup FEN, although you can get that one by just selecting the variant without an engine loaded and copying the current position). Apart from that, Sjaak actually doesn't "know" what a pieceToChar table is. All it knows are a startup FEN and possibly a "setup" string that have to be sent to XBoard.

It just seems to me that "knowing" these things requires knowing a great deal about understanding something about the inner workings of XBoard. From my perspective, it shouldn't be necessary to understand the intricacies and quirks of one particular GUI beyond what's specified in the protocol.

On the other hand, XBoard is clearly the only game in town when it comes to this particular niche, so it's not too bad to just target XBoard specifically. It also does a fantastic job of what it does, especially considering that it was never designed for playing half of what it can do now.

As another example of what I'd call an XBoard quirk that's not very well documented, I just got bitten (again) by the problem that if you specify holdingsize 1, you can only store pawns there. I realised what the problem was because I remembered you mentioned you needed to change the pieceToChar table for Mini Shogi.

[hgm]

Sure, I know that the current variant is MyAwesomeNewChessVariant357, because I defined that in my options list, XBoard placed it on a button and the user selected that, which prompted XBoard to send me "variant MyAwesomeNewChessVariant357". It's now up to me to tell XBoard what that means.
If I understand correctly, that would require me to keep track of whatever the previously selected variant was, ...

No. In this case the variant is unknown/invalid/engine-defined, and leaving it like that will cause an error for sure. Another way of thinking about it is that the variant is 'undefined' at this point, so that anything can happen when you leave it like that.

... as well as understand what that means in terms of pieceToChar - which is tricky because these aren't actually documented anywhere as far as I can tell

If the variant is not defined, the pieceToCharTable isn't either.

(this would be a good thing to have in the documentation, along with the startup FEN, although you can get that one by just selecting the variant without an engine loaded and copying the current position). Apart from that, Sjaak actually doesn't "know" what a pieceToChar table is. All it knows are a startup FEN and possibly a "setup" string that have to be sent to XBoard.

A pieceToCharTable has two aspects: the letters that occur in it, and the position of these letters in the table. The latter has no meaning for the engine, and is a pure GUI thing. (Except that in the case of + and ~ the engine should be able to match the promoted with the unpromoted pieces.)

It is true that this isn't documented anywhere, but in most cases it is pretty obvious (if you know the variant). And even if there is controversy over what the standard is, you can fire up XBoard, select the variant, select 'Copy Position', and you get the FEN of the opening position. Which in almost all cases contains all participating pieces (or at least all participating pieces that have letters associated with them). So in a sense XBoard is self-documenting here.

It just seems to me that "knowing" these things requires knowing a great deal about understanding something about the inner workings of XBoard. From my perspective, it shouldn't be necessary to understand the intricacies and quirks of one particular GUI beyond what's specified in the protocol.

That is debatable. The rules of Chess, for instance, are not specified in the protocol. Yet if you would not know how to play Chess at all, you would not get very far writing an engine for it, even if you are allowed to consult the CECP specs as often as you like. And knowing the names of the pieces or their ID can be considered (a very minor) part of the game rules.

The information that is in the ordering (i.e. which images to associate with a piece of a given name) is indeed a very troublesome aspect of the pieceToChar info. It is totally GUI dependent, and except for the orthodox Chess pieces there are no standard symbols. In an ideal world the pieceToChar info would be superfluous, and all info could be sent as 'piece' commands. The GUI could interpret the piece move, and decide what symbol best fits that move. (Or decide that globally for the entire piece set, if multiple pieces would compete for the same symbol.)

On the other hand, XBoard is clearly the only game in town when it comes to this particular niche, so it's not too bad to just target XBoard specifically. It also does a fantastic job of what it does, especially considering that it was never designed for playing half of what it can do now.

As another example of what I'd call an XBoard quirk that's not very well documented, I just got bitten (again) by the problem that if you specify holdingsize 1, you can only store pawns there. I realised what the problem was because I remembered you mentioned you needed to change the pieceToChar table for Mini Shogi.

Well, holdings are another 'open nerve'. One can specify holdingsSize, but not what the holdings are used for. E.g. whether captured pieces should go into their own holdings, or (after color flipping) into those of the opponent. Or, indeed, not at all (as in Seirawan).

I started with internal piece numbers (the pieceToChar ordinals) mapping 1:1 on holdings positions, which is why I had to squirm to get Shogi working, as it required all the held pieces to be contiguous in the low part of the table. This is why the Shogi Lance is really a Queen. (But promoted pieces needed special representation anyway, not useful for other variants, so that was not a big deal.)

Now this has been improved by at least ignoring pieces not present in the pieceToChar. But it still means I need an unnaturally large holdings size in Seirawan, because it also needs space for PNBRQ, while it only wants to store H and E.

Perhaps the 'piece' commands can be used to improve the matter. They can indicate which pieces can be dropped, and for pieces that cannot be dropped no place needs to be reserved in the holdings.

[Evert]

No. In this case the variant is unknown/invalid/engine-defined, and leaving it like that will cause an error for sure. Another way of thinking about it is that the variant is 'undefined' at this point, so that anything can happen when you leave it like that.

Ok, that interpretation works too. And is less ambiguous than I had previously understood.

It might be a good idea to phrase it exactly like that in the spec.

A pieceToCharTable has two aspects: the letters that occur in it, and the position of these letters in the table. The latter has no meaning for the engine, and is a pure GUI thing. (Except that in the case of + and ~ the engine should be able to match the promoted with the unpromoted pieces.)

It is true that this isn't documented anywhere, but in most cases it is pretty obvious (if you know the variant). And even if there is controversy over what the standard is, you can fire up XBoard, select the variant, select 'Copy Position', and you get the FEN of the opening position. Which in almost all cases contains all participating pieces (or at least all participating pieces that have letters associated with them). So in a sense XBoard is self-documenting here.

As you say, there are two aspects to this. It's not quite true that the ordering in the pieceToChar table is a pure GUI thing, at least not in the sense that I can ignore the problem completely. Things would break badly if I sent something like "PNBRQ..CAKpnbrq..cak", with A the Archbishop and C the Chancellor. Sure, XBoard would accept it, but the program forfeits the game when it wants to move either piece differently from a knight jump. Never mind what would happen if two engines had different pieceToChar tables.
What if I'm a bit OCD and insist that pieces should be listed alphabetically and send it ABCKNPQRabcknpqr? In short - I do need to know what order to put pieces in, and how many places to skip between them (or actually, have the full thing written out somewhere and override the appropriate entries, which is what I usually do).

In fact, I've had exactly this problem in the past, where I'd defined the Makruk pieces "Ferz" (F) and "Elephant" (E). In fact, I had Shatranj defined as using F and E as well. This works fine as long as you're not trying to setup a position, but of course breaks things when it comes to promotion.

Shatranj is an interesting problem (for historical reasons, I know) in that XBoard gives the startup FEN as "rnbkqbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBKQBNR w 0 1", but Q is very clearly not a queen and B is not a bishop (well, I suppose they actually could be, if XBoard still just has an "if (shatranj)" test; does it?). Either way, I either can't tell what positions in the pieceToChar for that variant are used, or I can't tell what movement-type actually corresponds to a particular piece type.

Clearly there are some things here that we would do differently (or at least more consistently) if we were building the thing up from scratch today, but we're not. The best that can be done, I think, is to document these things.

[Evert]

Ah, I see I was replying to the first part of your message while you were replying to the second part of mine.

That is debatable. The rules of Chess, for instance, are not specified in the protocol. Yet if you would not know how to play Chess at all, you would not get very far writing an engine for it, even if you are allowed to consult the CECP specs as often as you like. And knowing the names of the pieces or their ID can be considered (a very minor) part of the game rules.

Sure. But the rules of chess also don't tell me I have to list pieces in the order PNBRQK when telling a GUI that I want to use them.

I guess my main point was that this isn't documented very well, or specified in the protocol - which makes it feel like writing code targeting XBoard, rather than code that follows a protocol that could conceivable work in other GUIs that implement it (as I said, there's really no other GUI that can do what XBoard can, so the point is moot in a way).

In an ideal world the pieceToChar info would be superfluous, and all info could be sent as 'piece' commands. The GUI could interpret the piece move, and decide what symbol best fits that move. (Or decide that globally for the entire piece set, if multiple pieces would compete for the same symbol.)

I agree that the "piece" command would be the natural way to indicate what graphical representation to use (if we were designing the system from the ground up). Not sure about using the piece move though, that seems quite fragile. What if the move string is written in such a way that XBoard doesn't recognise that it represents a known piece? Betza notation is fuzzy, which has its advantages but also its disadvantages. Worse, what about pieces it doesn't know? What is a good metric for "best fit"?
Perhaps a combination of move-type and a hint by the engine for what might be a good representation would be best, but it's hard to work out how to do that exactly.
Ultimately, having a few well-defined pieces, plus some generic wild-cards is probably the most practical - but that's basically what we have already.

Well, holdings are another 'open nerve'. One can specify holdingsSize, but not what the holdings are used for. E.g. whether captured pieces should go into their own holdings, or (after color flipping) into those of the opponent. Or, indeed, not at all (as in Seirawan).

Yeah, that's the other issue: the game is supposed to start with some pieces in holdings, to be dropped on the board, and once they're gone, they're gone. But if I use variant 8x8+2_normal (or fairy), captured pieces end up in the holdings as well, even if I don't want them.

Now this has been improved by at least ignoring pieces not present in the pieceToChar. But it still means I need an unnaturally large holdings size in Seirawan, because it also needs space for PNBRQ, while it only wants to store H and E.

Perhaps the 'piece' commands can be used to improve the matter. They can indicate which pieces can be dropped, and for pieces that cannot be dropped no place needs to be reserved in the holdings.

That, or just send another "pieceToHoldings" string (if we're keeping to the same spirit as the existing design) which would default to pieceToChar, but could be used to eliminate unwanted pieces.

… in fact, that doesn't sound like such a terrible idea. Might even be feasible.

[hgm]

The reason I have been tardy providing unambiguous documentation for the pieceToCharTable is that I really feel we should be able to live without one completely. (And in the history of its existence the ordering of the pieces has already been changed several times. Chu Shogi actually required another change, but by now this would break too many other software, so I had to resort to the kludge of 'doubling the row length', and making some image substitutions in the old pieces specific to variant chu.)

It would be really nice if XBoard had a large, unstructured collection of piece images, with associated move sets. And then would try to find the best matches between available pieces and the definitions in the piece commands. The piece commands already provide the info which piece is the promoted version of which other (for Shogi promotions, which basically is the only case where non-Pawns promote). Then you could do completely without pieceToCharTable.

[Edit] I think the matching problem would be solvable.

[hgm]

What I had in mind for the move matching is to make a weighted count of atoms they have in common, (where R and B would be considered atoms), with a discount for missing directions. (I imagine that the pool of pieces kept by the GUI would almost all be completely symmetrical pieces.) The rarer atoms (in the pool as a whole) would get higher weights. So when there are only few pieces in the pool that have a Camel move, it would get a high priority to match a piece that does have the Camel moves with one of the Camel-like images.

Moves occurring on one and not on the other would also cause a penalty. This also should be proportional to the weights.