Formalizing the Universal Chess Interface

[hgm]

That's the ideal case. In practice, see the posting with the race conditions.

This has nothing to do with race conditions. Race conditions do not alter the order in which commands arrive at a given destination. They only apply to the timing of traffic in different directions.

[hgm]

I think this is a really valuable goal. As someone completely new to engine protocols (but not communications protocols in general) I found the UCI spec to be ambiguous in places. It seems like I've read through it 100 times and I always come away with more questions. The grammar specified for commands (and replies) should be unambiguous and context free. Nailing down the state chart for this process will be hugely beneficial for anyone implementing the protocol. So big THANKS from me!

Better wait with that until after you have discovered the hard way that this was just a recipe for how to not make your engine work on UCI-compliant interfaces...

[Ras]

This has nothing to do with race conditions. Race conditions do not alter the order in which commands arrive at a given destination. They only apply to the timing of traffic in different directions.

Please see my first posting in this thread where I explained what exactly I was referring to.

[hgm]

I think it is pretty obvious that all messages should be handled sequentially, unless explicitly specified otherwise.

One of my goals is to remove the need for subjective judgments, whether obvious or not, because obviousness varies from person to person and sometimes intuitive interpretations fall apart in unusual circumstances.

Well, very nice. But then just supplement the official specs with a few sentences stating the obvious. But don't mislead people in a way that would cause their engine to fail.

For example, you wrote that "[e]ach 'isready' will have to be answered with a 'readyok' when its turn to be processed from the input stream comes up." HMK does not mention queues, streams, or pipes. Although the engine's stdin and stdout will likely be pipes, it's not explicit that the engine should handle some messages as soon as they are read and divert some messages to a queue, and that messages cannot be coalesced even if the stated purpose of the message would still be accomplished edit: this is incorrect; the proper model seems to be that all messages are queued but some are contextually nonblocking (e.g. in between a go/stop pair, `isready` is nonblocking, but after stop or between stop and go, it is blocking), where "nonblocking" means "requiring immediate response" and "blocking" means "causing message processing to be paused until a moment at the engine's discretion" (that is, contingent on some internal state of the engine).

Sorry, but this is all complete nonsense. Stdin and stdout are streams. They can only be read in sequential order. Whether the input is a pipe or not is also irrelevant; when it would be the keyboard the program also doesn't get to see what you typed last before it has read away what you typed first. Engines cannot execute commands that they have not read yet. The engine is not in any position to decide what the "stated purpose of the message" would be; Clients can send commands for purposes only known to themselves. The blocking/non-blocking claims are simply false. Between 'go' and 'stop' an 'isready' can also be blocking: the 'go' might take some time to set up the search (creating and initializing search threads, clearing the hash table), and the 'isready' should not be replied to before it is done with that and ready to receive and execute a 'stop'. Why is it so hard to believe that an engine should reply to 'isready' only when it is ready?

During search the only allowed commands are 'stop' or 'ponderhit'.

This is not stated in HMK, in fact! (For reference, I'm looking at the copy that people can download here.)

Well, what other commands did you have in mind, then? There aren't that many GUI->engine commands, and 'setoption' is the only one that makes sense in this context. And for that the specs say: "this will only be sent when the engine is waiting.". Is there any doubt that an engine that is calculating/searching is not waiting?

In fact the specs do explicitly state that a 'debug' command can come while thinking. So I should add that to the list of allowed commands. Never any reason to defer its execution, though, as it is obviously only intended to set some flag that controls the amount of output the engine produces during thinking. So it can and should always be executed immediately. If it is implemented at all; otherwise it is just ignored.

It seems to me that 'uci' is only "supposed to come" at startup, and "register" only on the engine's request. That only leaves 'ucinewgame', 'position' and 'go', which would all be highly disruptive to a search in progress. (Or isn't that obvious enough?)

After receiving 'stop' the engine is by definition no longer searching.

But "searching" isn't defined in HMK!

It's actually a term I avoided using, because there may be some engines that don't perform a search but simply return the top recommendation of a policy network.

Well, I used it as synonymous for 'calculating' that is used in the specs. If you want to state the obvious, just add the sentence "calculating is what the engine does between executing the 'go' and the 'stop' command, or its unprompted production of a 'bestmove' output".

This isn't a confusion on my part; it was an attempt to point out that the specification never defines "calculating". The difference between calculating and wasting time cannot be part of a protocol, because protocols are merely rules of communication and cannot refer to the internal state of the engine (because internal representations may differ from engine) or the host device (is the engine calculating when its processor usage is above 10%? above 5%? and so on). So you could infer that "calculating" must be defined in terms of messages sent, but this is contradicted by the choice of term; "calculating" strongly suggests some internal condition of the engine. The same goes for the term "searching".

It should be obvious that executing commands takes non-zero time, even when it is only very little. And that sometimes it can be a lot (e.g. loading EGT). In the specs not all such time usage is considered 'calculating', however. The latter term is reserved for what the engine does to find a move in response to a 'go' command.

This is why the model currently used in the draft does not attempt to define the state of the engine or client; state is a global property and the transition rules are designed to ensure consistency regardless of the time between when messages are enqueued and when they are dequeued. (This is mentioned in the comment beginning at the bottom of page 5.)

In real life the state is a product of client and engine state, and both sending and receiving messages causes state change. If you are ignoring that you are certainly not talking about a global property. It seems to me that you are talking about the state of just one of the agents, which happen to have the same state diagram. But that for the engine the transitions occur when incoming messages are received, and outgoing ones sent, while for the client transitions it is just the other way around.

To be clear, I'm not trying to be cantankerous or intentionally obtuse, and I'm not trying to argue that any of your statements about the interface Stefan had in mind are false! I'm just concerned about anyone writing a client or engine who doesn't have access to an H. G. Muller to answer their questions ^_^

Well, I am nearly always here, so chances that people have access to me are probably a lot larger than that they would have access to any document you write.

[hgm]

This has nothing to do with race conditions. Race conditions do not alter the order in which commands arrive at a given destination. They only apply to the timing of traffic in different directions.

Please see my first posting in this thread where I explained what exactly I was referring to.

It seems to me this is just a problem caused by clearing the 'ignore input flag' and sending bestmove in the wrong order. Like you say, the 'ignore input flag' should never have an effect when dealing with a compliant client. So the risk that you would receive a position-go that would have had to be ignored by clearing the flag early is virtually non-existent.

[Ras]

It seems to me this is just a problem caused by clearing the 'ignore input flag' and sending bestmove in the wrong order.

The observed problem arises when doing that in the seemingly obvious order: send bestmove, then clear the flag. But doing it the other way around also has its own race condition, and that's non-trivial:
- clear ignore flag
- send bestmove
- engine search gets scheduled away before actually exiting / finishing
- next go comand arrives, new search is spawned
- old search threads get re-scheduled and do final cleanup that may conflict with the other, freshly spawned search.

My conclusion was that the easiest and most reliable way of avoiding any sort of that problem is simply not having such an ignore-flag to begin with. You won't notice such issues with a slow GUI such as Arena, but testing with the fast and unforgiving c-chess-cli while using all logical cores will run into such problems easily. I've seen that with several engines.

The only loophole I have is when a hash table resize arrives during search which will be queued up, but since that's a non-trivial command that may take time, the input thread will wait for confirmation from the worker (main) thread. That won't come as long as that thread is in search. Hence, the input thread uses a 5 seconds timeout and will just continue without confirmation so that the input thread will answer subsequent isready instead of blocking. The hash table resize will be executed after the current search ends.

Like you say, the 'ignore input flag' should never have an effect when dealing with a compliant client.

But if I am free to assume that the GUI will be compliant, then buffering the messages will not cause any functional non-compliance, either, because the whole issue revolves around tight timing of correct command sequences.

My engine actually pushes these cases accidentally because I have several conditions for "quick responses" using nearly no thinking time so that for the other engine, the time between sending its bestmove and receiving the next position command can be extremely short. I'm also bypassing the whole C standard library for extra I/O performance, geared towards games at 1s/game with no increments.

[expositor]

I think this is a really valuable goal. As someone completely new to engine protocols [...] I found the UCI spec to be ambiguous in places. It seems like I've read through it 100 times and I always come away with more questions.

Thank you for the encouragement! I appreciate it.

It will be several months (and several more drafts) before anything is finalized, and widespread adoption of the new standard may take several years (if it happens at all), so I'm afraid this may not be very helpful for your immediate work of writing a client. But I hope it'll be helpful for future work!

In real life the state is a product of client and engine state, and both sending and receiving messages causes state change. If you are ignoring that you are certainly not talking about a global property. It seems to me that you are talking about the state of just one of the agents, which happen to have the same state diagram.

The model is explained in section 3 of the draft, so you don't have to guess what I mean.

If you find any bugs, let me know! I'm aware of one snafu that the author of Frozenight brought up:

MinusKelvin Small bug: engine sending `bestmove` in the `ping` state needs to transition to the `sync` state rather than the `idle` state. Otherwise, you allow `isready` without a matching `readyok`.
Kade I actually hadn't thought to try to ensure that every `isready` is matched by a `readyok`. My thinking was that `isready` could be answered with `bestmove`, so that all three of these would be valid:
1: Client sends isready → engine sends bestmove → engine reads isready → engine sends readyok
2: Client sends isready → engine reads isready → engine sends readyok → engine sends bestmove
3: Client sends isready → engine reads isready → engine sends bestmove
But yes! I agree that if we wanted to ensure that every `isready` is matched by a `readyok`, there'd need to be a transition to sync from ping. (And I think that'd be enough? although I've yet to work through all the cases.)

———

HGM, I've repeatedly stated (in this thread and in the draft itself) that there are intentional changes introduced by the draft. It is a revision of UCI as well as a formalization. Repeatedly explaining that UCI is obvious to interpret is orthogonal to either goal (and makes me wonder somewhat whether you bothered to read what I wrote).

If you're opposed to the existence of a project like this, I'd be happy to hear the reasons why, but that's a separate topic and I'd prefer to discuss it in a separate thread.

[hgm]

HGM, I've repeatedly stated (in this thread and in the draft itself) that there are intentional changes introduced by the draft. It is a revision of UCI as well as a formalization. Repeatedly explaining that UCI is obvious to interpret is orthogonal to either goal (and makes me wonder somewhat whether you bothered to read what I wrote).

If you're opposed to the existence of a project like this, I'd be happy to hear the reasons why, but that's a separate topic and I'd prefer to discuss it in a separate thread.

Well, then it is a completely pointless exercise. People are not going to abandon UCI for any new protocol you invent. Not in a million years.

If you try to present this as UCI, and thereby mislead engine authors into thinking they can follow your specs to make their engine interoperable with true UCI software, you should not be surprised if others (and I in particular) will warn them against such practice.

[hgm]

The observed problem arises when doing that in the seemingly obvious order: send bestmove, then clear the flag. But doing it the other way around also has its own race condition, and that's non-trivial:
- clear ignore flag
- send bestmove
- engine search gets scheduled away before actually exiting / finishing
- next go comand arrives, new search is spawned
- old search threads get re-scheduled and do final cleanup that may conflict with the other, freshly spawned search.

Umm, I would consider clearing the flag after sending bestmove the obvious way not to do it, because it invites the race condition: after bestmove you can get commands that should not be ignored, and there is no guarantee that the flag will be cleared before you receive them. And for the example of the opposit case: this would also be avoided by sending bestmove only after all threads have cleaned up. I guess doing otherwise would be cheating anyway: in a non-ponder game engines are not supposed to do anything other than waiting for input in the opponent's turn, which starts as soon as they send bestmove.

But if I am free to assume that the GUI will be compliant, then buffering the messages will not cause any functional non-compliance, either, because the whole issue revolves around tight timing of correct command sequences.

Perhaps not, but it is totally needless code complexity. If you would only send bestmove after the search has cleaned up, you would not have to do any such thing, and be still resistant to non-compliant GUI behavior. (Not that the latter would help you getting sensible results under a rogue GUI...)

[Ras]

Feedback would be enormously appreciated; I'm particularly interested in comments from engine devs

1. One of the strong points of UCI is that it isn't amended all the time, resulting in a stable ecosystem instead of fragmentation. UCI engines simply tend to work. I don't see the need for a mostly identical but slightly incompatible UCI version.

2. I don't like the state diagrams. That's a possible the implementation of a protocol - the protocol definition should not be concerned with implementation. It's only relevant what goes back and forth "on the line".

3. I find the original spec easier to understand.

4. What would be good would be the old spec, just with some more clarifying comments on frequently asked questions and hints for how the ecosystem has developed in reality. For example, most UCI engines including Shredder cannot deal with parameters after searchmoves in go if searchmoves isn't the last parameter. Similarly, the PV should probably be the last part of the info message during search. Everyone does it that way already, but that should be documented.