Formalizing the Universal Chess Interface

[expositor]

I don't think this is a correct description of UCI as it is in common use.

That is correct! There are intentional differences – the Revision Notes section mentions some of these.

The current draft will require most clients to change, which may be too much to ask. My rationale was roughly:

1. Despite having read HMK, I've never managed to get the UCI interface quite right. Every time I've tried to get Expositor to work with a different client (cutechess, lichess-bot, external-engine, liground, ...), I've needed to fix something or introduce a hack, and I've seen this happen to another engine dev as well. This suggested the need for a more comprehensive specification.
2. Unifying the behavior of clients and engines may mean that some clients and engines will need to change (this is unavoidable). We may also want to change the behavior of clients or engines to have tidier solutions for some questions (this is merely convenient).
3. Since there are many more engines than clients, it'd perhaps be better to change existing clients than to change existing engines.

The problem is a bit less severe than I'm presenting – my observation has been that in practice UCI isn't defined by HMK but is defined empirically (by what Stockfish does, or Cutechess, or a few popular clients and engines), and that resolves some questions.

The 'isready' command exists purely for the convenience of the client; [...] the engine doesn't derive any rights from it (and in particular not the right to receive no input).

The problem I see with HMK is that it leaves so many questions unanswered (which I've always found to be rather frustrating). Most of the ambiguity has to do with mixing synchronous and asynchronous interaction. Here a few questions, for example:

• What should happen when the client sends "isready\n" twice in a single write, or twice before the engine has a chance to read the pipe? is one `readyok` sufficient, or does the engine need to send two replies? (Perhaps the underlying question is, are some messages meant to be handled sequentially, in a blocking manner, and some messages are meant to be handled as soon as they are read, skipping ahead of "sequential" messages?)
• HMK states that `isready` is used both "to wait for the engine to be ready again" and "to ping the engine to find out if it is still alive". When is it the former and when is it the latter? If the client sends `stop` and then `isready` in immediate succession, and the engine is still calculating, once the engine reads the message should it wait to answer with `readyok` until after it has stopped (because `isready` was sent after `stop`) or immediately (because "when the engine is calculating [...] the engine should immediately answer with `readyok` without stopping the search")?
• Can the engine assume that the client will never send `setoption` while it is calculating? That seems like a reasonable assumption, but HMK doesn't mention it. If it does (suppose the client sends `stop` and `setoption` in immediate succession, so that the engine recieves `setoption` while it is still calculating, before it has had time to stop, or perhaps the client simply sends `setoption` in the middle of a search), should the engine ignore the `setoption` or apply the change once it has stopped?
• HMK does not require the client to send `isready` before a search (only once at the beginning to "wait for the engine to finish initializing"). HMK also states that "if the engine receives a command which is not supposed to come, for example `stop` when the engine is not calculating, it should also just ignore it." By the same token, one might expect that the engine should ignore `go` if it receives `go` during a search. If the engine recieves a go message while it is still calculating (because the client sent `stop` and `go` in immediate succession and the engine read the go message before it had halted), should it ignore the go message? Or should it queue up an indefinite number of go messages? Or just one? (Perhaps the underlying question is, for the purposes of UCI, is the engine's current state defined by client messages (either defined by client messages being sent or defined by client messages being read), or defined by both client and engine messages, or is the engine's state defined by unobservable, internal variables?)

And I generally tried to resolve these sorts of questions in the simplest way I could think of. Admittedly, there was the intentional bias I mentioned: I generally picked solutions that simplified the implementation of engines.

I had tried modeling the interaction between client and engine as a collection of events (client writes to a pipe, engine writes to a pipe, clients reads from a pipe, engine reads from a pipe) and happens-before relationships, but the diagram got rather ugly rather quickly.

I usually think of message-passing between the client and engine as happening more or less instantaneously, but that's not actually the case, so the current model used by in draft was an attempt to present the client-engine system as having a single, coherent global state (rather than a disjoint pair of client and engine states) while still being consistent and technically correct.

There is probably a better way to model it, though.

Some existing clients in fact send stop + position + go ponder simultaneously.

This had crossed my mind, and one of the Outstanding Issues is the question Should the client be allowed to queue up messages in the halt state? I'm not entirely sure how to reconcile this with the current model, where transitions occur when messages are sent, which is perhaps another indication that it needs some amount of rethinking.

And I still need to get around to incorporating pondering :/

Your state diagram also ignores that 'go' is illegal without a preceding 'position'.

This was one of the intentional changes, meant primarily to simplify the state+transition model (and secondarily to reduce the surface area of undefined behavior). In such cases, I generally tried to match the behavior of common engines (Stockfish and Komodo, for example).

[expositor]

Why do you specify UTF-8 and not ASCII, when you only ask for a subset of ASCII?

That's a good question! I hadn't really thought about it, but it allowed client messages to also be described using the terminology of Unicode scalar values and tokens (themselves defined as sequences of Unicode scalar values).

H. G. Muller's answer is good. And it highlights the use of "scalar values" rather than "codepoints", because the latter includes surrogate pairs (an artifact of UTF-16) while the former does not.

In any case, this will definitely change. The author of Frozenight brought this up in another discussion:

MinusKelvin 1.3: ASCII requirement precludes placing my syzygy tablebase files in a folder called `échecs` (french for chess), so this should probably be adjusted to match 1.4

Kade Will do. In fact, you've pointed out what is simply a mistake – as written, it's possible for the engine to name an option that the client cannot use. I'm still not sure what the best way is to handle file paths, which on many systems are just a series of bytes (with only a few restrictions). Perhaps the best thing is relaxing the UTF-8 requirement for anything following `info string` and `option name ... value` (but still disallowing U+000A and U+000D, which seems like a reasonable restriction).

[Ras]

And why not everything including tokens?

The FEN string uses case for distinguishing between piece colours, and path names (e.g. EGTBs) must be treated as case sensitive because there are file systems that are case sensitive. So you can't just normalise everything.

I guess there is also ASCII encoded as UTF-16 (wide char).

Even UTF-32 exists, but those representations are not used for data exchange, only program internally. Externally, it's UTF-8 which has the nice property that the usual C functions like strcpy, strcmp work transparently. The only difference is that the equation "one byte is one character" doesn't hold anymore.

the question Should the client be allowed to queue up messages in the halt state?

I do that because the GUI isn't allowed to send messages other than stop/isready during search anyway. The problem with discarding messages engine-side is that it opens bugs of wrongly discarding. I've seen a race condition in a number of engines with dedicated input thread and discarding that goes like this, usually when you run a tournament using all logical cores:
- Engine sends bestmove.
- Engine gets scheduled away.
- GUI sends next command.
- Engine input thread gets scheduled.
- Engine input thread discards command.
- Engine search / main / whatever thread gets scheduled and sets the "stop ignoring messages" flag for the input thread.
Result: if the engine ignores both the position and the go command, it will hang and lose on time while still reacting to isready. If only the position command is ignored, but the flag setting arrives in time to process the go command, the engine calculates on the wrong position, i.e. the one from the turn before, and may put out a nonsensical or even illegal next bestmove.

The quick and dirty fix is to first set the "accept messages" flag for the input thread and then printing the bestmove, but at that point, it's another race condition that would hit in if the GUI were to send a message between setting the flag and printing the bestmove. Not a problem in practice because GUIs don't do that, but then what's the point of having that ignore flag to begin with?

So my solution is that my input thread always reads out the arriving messages and processes stop/quit/isready immediately, but puts other messages in a queue that will be processed once the search ends. If I don't discard messages, I'll never erroneously lose some.

[hgm]

What should happen when the client sends "isready\n" twice in a single write, or twice before the engine has a chance to read the pipe? is one `readyok` sufficient, or does the engine need to send two replies? (Perhaps the underlying question is, are some messages meant to be handled sequentially, in a blocking manner, and some messages are meant to be handled as soon as they are read, skipping ahead of "sequential" messages?)

I don't think there is any ambiguity here. Each 'isready' will have to be answered with a 'readyok' when its turn to be processed from the input stream comes up. I think it is pretty obvious that all messages should be handled sequentially, unless explicitly specified otherwise. Any protocol would degenerate to a total mess when commands would be allowed to be processed in an arbitrary order.

The exception that is explicitly made states that 'isready' must be replied to even when searching (which would be perceived out-of-turn by those who interpret the search and printing of 'bestmove' as processing of the 'go' command).

HMK states that `isready` is used both "to wait for the engine to be ready again" and "to ping the engine to find out if it is still alive". When is it the former and when is it the latter? If the client sends `stop` and then `isready` in immediate succession, and the engine is still calculating, once the engine reads the message should it wait to answer with `readyok` until after it has stopped (because `isready` was sent after `stop`) or immediately (because "when the engine is calculating [...] the engine should immediately answer with `readyok` without stopping the search")?

HMK is Stephan?

I also see no ambiguity there. It is for what the client wants to use it for. The protocol specs are not a tutorial for how to develop clients. The specs just say how the comment should be responded to by an engine: 'readyok' indicates the engine is ready to process any allowed incoming command immediately. The client can use this for any purpose it sees fit. During search the only allowed commands are 'stop' or 'ponderhit', and the engine should always be ready to start processing those without delay. After receiving 'stop' the engine is by definition no longer searching/calculating. If it needs time to kill its search threads, during which it is not ready for processing (as opposed to receiving!) the next command (presumably a position-moves), it should defer the 'readyok' (replied to an 'isready' it received after the 'stop' command) until it is.

Usually clients would not be interested to know how much time the engine uses here, though, as the time for winding down the ponder search would simply be counted against the clock of the engine, which would start to run when the client send the 'go' command for the next move. (And not when it pleases the engine to start that search).

Can the engine assume that the client will never send `setoption` while it is calculating? That seems like a reasonable assumption, but HMK doesn't mention it. If it does (suppose the client sends `stop` and `setoption` in immediate succession, so that the engine recieves `setoption` while it is still calculating, before it has had time to stop, or perhaps the client simply sends `setoption` in the middle of a search), should the engine ignore the `setoption` or apply the change once it has stopped?

As far as the client is concerned an engine is only 'calculating' between the 'go' and the 'stop' command. So yes, clients can send stop + setoption in immediate succession. (The need to set the option could actually have been the reason for sending the 'stop' during pondering, so this is in fact a likely scenario.)

HMK does not require the client to send `isready` before a search (only once at the beginning to "wait for the engine to finish initializing"). HMK also states that "if the engine receives a command which is not supposed to come, for example `stop` when the engine is not calculating, it should also just ignore it." By the same token, one might expect that the engine should ignore `go` if it receives `go` during a search. If the engine recieves a go message while it is still calculating (because the client sent `stop` and `go` in immediate succession and the engine read the go message before it had halted), should it ignore the go message?

Of course not. It is not a command that "is not supposed to come". You confuse 'calculating' with "wasting time for internal reasons". As soon as an engine receives 'stop', it is no longer 'calculating' in the sense of the protocol, and whatever it is doing is just "processing of the previous command (= stop)". There is no rule that you cannot send commands before processing of a previous command finishes. If there was, clients would have to poll the engine with 'isready' and wait for the 'readyok' before every command other than 'go'.

Or should it queue up an indefinite number of go messages?

Of course it should queue up an indefinite number of commands. Not doing that would be tantamount to randomly ignoring commands from the input stream just because they happened to arrive at a moment when the engine was still working on processing a previous command. Clients in general would not know when that occurs. Normally you would not even have to think about this, because the pipe mechanism in the operating system does it. All commands are queued in the pipe, and the engine reads them away one by one, and processes them as soon as it reads them. It will be impossible for the engine to determine how long a command has been waiting in the pipe.

Or just one? (Perhaps the underlying question is, for the purposes of UCI, is the engine's current state defined by client messages – either defined by client messages being sent or defined by client messages being read –, or a combination of client and engine messages, or is the engine's state defined by unobservable, internal variables?)[/list]And I generally tried to resolve these sorts of questions in the simplest way I could think of. Admittedly, there was the intentional bias I mentioned: I generally picked solutions that simplified the implementation of engines.

As clients and engines cannot know each other's timing of events, I don't see how it could be any other than that, as far as the client is concerned, the state of the engine is determined by the commands the client sends, while for the engine the state is defined by the commands the engine receives. It seems to me that any other interpretation would need violation of the laws of nature, and thus is probably not what was meant.


I don't see how the existing specs leave any of these issues undefined. But I see how your description (in particular the state diagram) conflicts with the original specs. So it is just not a description of UCI.

[hgm]

So my solution is that my input thread always reads out the arriving messages and processes stop/quit/isready immediately, but puts other messages in a queue that will be processed once the search ends. If I don't discard messages, I'll never erroneously lose some.

This sounds wrong, because it opens the possibility for 'isready' to be processed out of order. While its very purpose could have been to test whether all preceding command (which you have in fact queued) have been processed.

I also don't see why you would have to do any explicit queueing. The simple implementation would be to just read the commands one by one from stdin, and only start reading the next command when the engine is done processing the previous one. This would happen naturally for commands processed by the input thread itself (like 'setoption' and such). The only case I see where you would have to wait for an internal event is after 'stop', where you might want to wait for the search threads to acknowledge that they indeed have all stopped. (This would usually be after one of them had printed 'bestmove', even though that is not strictly necessary.) So then you simply do not read input until this is acknowledged.

You might argue that this would make it impossible to react to a 'quit' command when the search fails to properly stop. But this indicates a crash, and it is questionable whether the engine would be able to properly execute a 'quit' command in that case. If you worry about that you could have the input thread apply a timeout to its waiting for the stop acknowledgement, and quit the engine spontaneously (perhaps with an error message 'fails to stop').

[Ras]

This sounds wrong, because it opens the possibility for 'isready' to be processed out of order. While its very purpose could have been to test whether all preceding command (which you have in fact queued) have been processed.

That's why I have additional logic for commands that may not be finished instantaneously: ucinewgame and changing the hash table size. That's because ucinewgame clears the entire hash table while reallocating it not only does calloc, but also forcefully writes data and then clear the tables again. That forces the OS to actually blend in the pages instead of just having them mapped to the zero page and doing the remap later in search at the expense of engine performance.

I also don't see why you would have to do any explicit queueing.

Because isready has to be answered at all times, also during search, and the clean way is a dedicated input thread. So what if the engine has received the go command and is calculating? Process the commands and discard all that are not stop or isready? See my posting above why I don't do it that way.

[expositor]

I think it is pretty obvious that all messages should be handled sequentially, unless explicitly specified otherwise.

One of my goals is to remove the need for subjective judgments, whether obvious or not, because obviousness varies from person to person and sometimes intuitive interpretations fall apart in unusual circumstances.

For example, you wrote that "[e]ach 'isready' will have to be answered with a 'readyok' when its turn to be processed from the input stream comes up." HMK does not mention queues, streams, or pipes. Although the engine's stdin and stdout will likely be pipes, it's not explicit that the engine should handle some messages as soon as they are read and divert some messages to a queue, and that messages cannot be coalesced even if the stated purpose of the message would still be accomplished edit: this is incorrect; the proper model seems to be that all messages are queued but some are contextually nonblocking (e.g. in between a go/stop pair, `isready` is nonblocking, but after stop or between stop and go, it is blocking), where "nonblocking" means "requiring immediate response" and "blocking" means "causing message processing to be paused until a moment at the engine's discretion" (that is, contingent on some internal state of the engine).

HMK is Stephan?

HMK refers to the Description of the universal chess interface rather than Stefan himself. (The abbreviation is introduced on page 15 and used on pages 15 and 16).

During search the only allowed commands are 'stop' or 'ponderhit'.

This is not stated in HMK, in fact! (For reference, I'm looking at the copy that people can download here.)

After receiving 'stop' the engine is by definition no longer searching.

But "searching" isn't defined in HMK!

It's actually a term I avoided using, because there may be some engines that don't perform a search but simply return the top recommendation of a policy network.

You confuse 'calculating' with "wasting time for internal reasons". As soon as an engine receives 'stop', it is no longer 'calculating' in the sense of the protocol, and whatever it is doing is just "processing of the previous command (= stop)".

This isn't a confusion on my part; it was an attempt to point out that the specification never defines "calculating". The difference between calculating and wasting time cannot be part of a protocol, because protocols are merely rules of communication and cannot refer to the internal state of the engine (because internal representations may differ from engine) or the host device (is the engine calculating when its processor usage is above 10%? above 5%? and so on). So you could infer that "calculating" must be defined in terms of messages sent, but this is contradicted by the choice of term; "calculating" strongly suggests some internal condition of the engine. The same goes for the term "searching".

As clients and engines cannot know each other's timing of events, I don't see how it could be any other than that as far as the client is concerned, the state of the engine is determined by the command it sends, while for the engine the state is defined by the commands it receives.

This is why the model currently used in the draft does not attempt to define the state of the engine or client; state is a global property and the transition rules are designed to ensure consistency regardless of the time between when messages are enqueued and when they are dequeued. (This is mentioned in the comment beginning at the bottom of page 5.)

I don't see how the existing specs leave any of these issues undefined. But I see how your description (in particular the state diagram) conflicts with the original specs. So it is just not a description of UCI.

I'd contend that there is no such thing as UCI – or less dramatically, that the term UCI is descriptive, rather than prescriptive, referring to a collection of behaviors that several programs have rather than a precise protocol. I don't think UCI can be defined as "whatever Shredder does", because I don't believe this is what most people have in mind. What I suspect most people have in mind is "my engine supports UCI if it mostly works with cutechess or the 'UCI' clients that I use" or "my client supports UCI if it mostly works with Stockfish and other 'UCI' engines that I use" and so on. Which is a workable definition, but one that I think can be improved upon, because I've seen implementation divergence and confusion (myself, for one).

But this is beside the point, and you're entirely right: this formalization conflicts with the behavior of existing clients in a few ways (where legacy engines will work with new clients but legacy clients may not work with new engines). So maybe it should have a slightly different name.

———

To be clear, I'm not trying to be cantankerous or intentionally obtuse, and I'm not trying to argue that any of your statements about the interface Stefan had in mind are false! I'm just concerned about anyone writing a client or engine who doesn't have access to an H. G. Muller to answer their questions ^_^

[hgm]

Because isready has to be answered at all times, also during search, and the clean way is a dedicated input thread. So what if the engine has received the go command and is calculating? Process the commands and discard all that are not stop or isready? See my posting above why I don't do it that way.

Isn't that what the UCI specs prescribe? "if the engine receives a command which is not supposed to come, it should also just ignore it." During search other commands than isready, stop or ponderhit are not supposed to come.

So how is queueing them anything other than non-compliance?

And 'isready' doesn't have to be answered at all times. It has to be answered when all commands received before it have finished processing. That is the entire point of sending 'isready' after setting the options at startup; options like Hash or tablebase loading might take a long time, and the client doesn't want to start the engine's clock before the latter has finished that. Replying with 'readyok' before that time would defeat the purpose.

[Ras]

Isn't that what the UCI specs prescribe? "if the engine receives a command which is not supposed to come, it should also just ignore it." During search other commands than isready, stop or ponderhit are not supposed to come.

That's the ideal case. In practice, see the posting with the race conditions.

So how is queueing them anything other than non-compliance?

That's a non-compliance without functional impact as long as everything works as it should. The risk for race conditions and erroneous discard would have functional impact and be even less compliant.

And 'isready' doesn't have to be answered at all times.

Yeah, with or without search, and in particular during search, that's what I meant.

Replying with 'readyok' before that time would defeat the purpose.

Correct. Please read the posting that you just replied to.

[JoAnnP38]

To be clear, I'm not trying to be cantankerous or intentionally obtuse, and I'm not trying to argue that any of your statements about the interface Stefan had in mind are false! I'm just concerned about anyone writing a client or engine who doesn't have access to an H. G. Muller to answer their questions ^_^

I think this is a really valuable goal. As someone completely new to engine protocols (but not communications protocols in general) I found the UCI spec to be ambiguous in places. It seems like I've read through it 100 times and I always come away with more questions. The grammar specified for commands (and replies) should be unambiguous and context free. Nailing down the state chart for this process will be hugely beneficial for anyone implementing the protocol. So big THANKS from me!