Formalizing the Universal Chess Interface

[Fulvio]

There is a problem with that definition, because it delimits the "search mode" (which the specs call "calculating") by communication events in opposit directions. This can lead to race conditions.

None come to my mind.
The main point is that the engine can send messages only in "search mode" (otherwise it is only allowed to reply).
So, if the messages are processed in order as you correctly mentioned, no race condition are possible before sending "go".
In "search mode" maybe "stop" and "bestmove" can be sent at the same time, but that's specified (the engine should just ignore it).
The way ponder is implemented is ugly, but the engine cannot send "bestmove" before receiving "ponderhit".
Can you make an example?

A 'go' command will bring the engine in the "calculating" state, during which it is not allowed to send it any other commands than 'stop', 'ponderhit', 'debug' and 'isready' (and 'quit'?). This limitation lasts until the 'stop' command or the 'bestmove' command (whichever comes earlier).

Some terms like "calculating", or "infinite", are confusing.
It is natural to think that if the engine is non calculating it should leave "calculating mode".
There are explicit statement, like "Do not exit the search in ponder mode, even if it's mate!", to correct that (https://www.shredderchess.com/download/div/uci.zip)
But everything is just simple and obvious if you specify that the client is asking the engine to find the best move and when it wants the response.

[expositor]

Any chance you could detail some specific examples? I think aside from the ongoing debate about this project it would be quite nice to compile a list of common problems/edge cases for beginners.

I've forgotten the details of the problems that I personally ran into, since it's been a while, but I think I remember what they were about. They were problems that my friend (another engine dev) also ran into, so I may be able to ask him.

Anyway, on two separate occasions, we were trying out a new client and by complete coincidence both had the same issue; on both occasions, the symptom was that the connexion would eventually hang. The problems had to do with some combinations of `stop` and `go` sent in immediate succession and with searches that finished instantly due to mate distance pruning. The cause of one problem ended up being a bug in our engines and the other was a bug in the client (which we had to write hacks around).

———

Maybe less words and more charts and diagrams. Perhaps a fully exploded grammar for the commands/responses would be nice as well. BNF would work well there.

Will do! I'll plan to include summaries using EBNF or PEG in the next draft or two.

———

Hey, I just wanted to chime in

thank you, Clayton! ^_^

If you remember any questions you had or difficulties you encountered, that'd be really helpful. (No obligation, though!)

———

Honestly, I didn't find it very formal.

True, it's not written in Coq, Lean, TLA⁺, or something similar, and I do think that properly the term formal should be reserved for that sort of thing. Describing it as "formal" bothers me a bit, but it's a convenient shorthand; by "formal" I just mean precise and complete.

If there are any passages that are imprecise, that aren't exhaustive, or inconsistently use terminology, please let me know!

If the engine sends an illegal move maybe it will lose the game. Or maybe the current position will be sent again with the updated time. Or maybe something else. It depends on the circumstances and there is no right behavior that needs to be specified.

Sure! but then either the behavior should be explicitly stated to be implementation-defined, or it should be explicitly stated that the event causes the governance of the specification to end.

It is impossible to guarantee that all limits are met, a fast computer may reach the desired depth in less than movetime. A slower one maybe not.
The engine should just stop thinking when it reaches the first limit.

My apologies – this was a bad example on my part! because it's actually outside the purview of a communication protocol. (However, a recommended interpretation is mentioned in the draft as a comment.)

———

As far as I am concerned almost anything would be better than UCI.

Oh, that's right! You wrote the second version of the Chess Engine Communication Protocol ;)

You missed the point. The recommendation was that there should not be any difference. Just clarification and recommendations. [...] Designing new protocols is something completely different from improving the specification of an existing one. Just call it NCI for New Chess Interface and you would not hear any objections from me.

Truthfully, I'm not sure what to do about the naming.

On one hand, it seems right to give it a separate name so that we can talk about HMK vs "NCI". On other hand, this is pretty obviously just a flavor of UCI, not a brand-new protocol.

Suppose the restriction that the client wait for `bestmove` after sending `stop` is already fixed and removed; that leaves only one incompability that could cause things to break if you used an "NCI" engine with a legacy UCI client: whether or not the engine is allowed to send `bestmove` befores it receives the `stop` following a `go infinite` message.

HMK says the engine may not. If HMK is the definition of UCI, then none of the following are UCI engines:

• Berserk
• Koivisto
• Ethereal
• Nemorino
• Velvet
• Halogen
• Asymptote
• Glaurung

That's because these engines all send a `bestmove` message (nearly instantaneously) in response to

Code: Select all

position fen 8/8/8/8/8/1QK5/8/k7 w - - 0 1
go infinite

However, everyone considers them to be UCI engines. They are in fact "NCI" engines! This suggests that, in actual usage, whatever the term UCI refers to very much includes "NCI". And in light of that observation, a name like UCI version 2 seems perfectly reasonable.

[jshriver]

Feedback would be enormously appreciated; I'm particularly interested in comments from engine devs and client devs (by client I mean a user interface or a utility that interfaces with engines, for example).

As Im work working on my own engine and have written a couple different analysis clients. One big thing that hit me recently is there is nothing in UCI to set side to move.

I'm using Arena now due to its UCI log viewer to see how it's done and seems the only way to do it is just
* set startpos
* send moves to that point
* go ...

Seems like there should be a clearer way of handling this.

[Ras]

I am fairly certain that, given the current draft, a conforming client would work with most (if not all) legacy engines.

Commercial engines are already out - the copy protection fell victim to the "nicer on the line" logic. Breaking the ecosystem just for "nicer on the line" is the wrong approach IMO.

By contrast, introducing a new info error message is not useful because the customary way it's done now is using info string, and that works with existing clients as well. So instead, this could be specified as info string error (message goes here). New clients could do something fancy with that, but old ones would simply display it in the console window (if any).

Introducing WDL instead of CP for NN engines sounds nice on the surface - but a lot of clients simply will not be updated, and stuff like that is bad for the ecosystem. Or there could be a two way feature negotiation, and at that point, things would go downhills like in CECP. IMO, you're too much concerned with making things nice for everyone - a core strength of UCI is exactly that it did not cater to everyone's wishes.

The range in the options from 0 to 2^63 - 1 is probably just a spec bug because negative numbers are required e.g. for configuring threshold values such as contempt.

Also, quote from the draft:

Although SPEC is designed so that conforming engines will be usable with nonconforming, legacy clients (and vice versa), some configurations may be incompatible.

That should not be the case unless we're talking about incompatibilities that already exist, i.e. no new ones are introduced.

I don't understand how it would qualify as a specification, then, rather than an informal tutorial or "getting started" guide.

Besides some content issues that would warrant more clarity, I didn't have trouble implementing that. But that's just a matter of a few sentences.

the client is not required to send a position before sending go

That may introduce a breaking change in case of repeated go: is the engine expected to keep the last transmitted board position, or will it execute the bestmove on its internal board? RN, this doesn't matter because there is no repeated go allowed anyway. Means, existing engines may behave either way.

But what's the use of that change with breaking potential? The "nicer on the line" thinking strikes again.

[Ras]

Seems like there should be a clearer way of handling this.

What's unclear about that? The side to move is set by giving the starting position (with White to move) or a FEN with STM included, then executing alternate moves.

[Fulvio]

HMK says the engine may not. If HMK is the definition of UCI, then none of the following are UCI engines:

• Berserk
• Koivisto
• Ethereal
• Nemorino
• Velvet
• Halogen
• Asymptote
• Glaurung

That's because these engines all send a `bestmove` message (nearly instantaneously) in response to

Code: Select all

position fen 8/8/8/8/8/1QK5/8/k7 w - - 0 1
go infinite

That's sad, but understandable. Many engines are an hobby and implementing the protocol is not the fun part.
So, what's the problem?
Let's assume we have the simplest GUI, just a board and a button engine on/off.
We have three possible action:
[start engine]
-> isready
wait_for readyok
-> position ...
-> go infinite

[stop engine]
-> stop
wait_for bestmove

[react to a user move on the board]
if the button engine is off -> do nothing
generate event stop engine
generate event start engine

Everything is simple, in sync and beautiful.
Even if the user makes multiple moves when we are waiting for bestmove, we simply do nothing and just send the last current board to the engine.

But we have this non-compliant engines, how we can deal with them?
The most common thing is: do not wait for bestmove and just avoid any synchronization. The pipes are buffered and usually the engines are really fast, so users would not notice the side effects (you may show pv lines that are not related to the current board. And if the engine is slow, it would analyze old positions even if the user already moved on).

But the question is why?
What are the advantages of this indecisiveness where the engine maybe unpredictably leave "search mode"?
Even when it is clearly specified: "Do not exit the search without being told so in this mode!"
I think chess players, myself included, have some compulsive behavior, and I really think that the root of the problem may be the term "calculating". Staying in that mode when the engine is not thinking seems wrong somehow and hunt them at the back of their mind.

[Ras]

Staying in that mode when the engine is not thinking seems wrong somehow and hunt them at the back of their mind.

My guess is that the authors don't care about that part of the spec because it's not important for engine matches.

Here's how my engine deals with that:

Code: Select all

position fen 8/8/8/8/8/1QK5/8/k7 w - - 0 1
go infinite
info depth 2 seldepth 2 score mate 1 time 1 nodes 15 nps 15000 hashfull 0 tbhits 3 pv b3b2
info depth 3 seldepth 3 score mate 1 time 1 nodes 17 nps 17000 hashfull 0 tbhits 5 pv b3b2
info depth 4 seldepth 4 score mate 1 time 1 nodes 19 nps 19000 hashfull 0 tbhits 7 pv b3b2
info depth 5 seldepth 5 score mate 1 time 1 nodes 21 nps 21000 hashfull 0 tbhits 9 pv b3b2
info depth 6 seldepth 6 score mate 1 time 1 nodes 24 nps 24000 hashfull 0 tbhits 12 pv b3b2
info depth 7 seldepth 7 score mate 1 time 1 nodes 27 nps 27000 hashfull 0 tbhits 15 pv b3b2
info depth 8 seldepth 8 score mate 1 time 1 nodes 30 nps 30000 hashfull 0 tbhits 18 pv b3b2
info depth 9 seldepth 9 score mate 1 time 1 nodes 33 nps 33000 hashfull 0 tbhits 21 pv b3b2
info depth 10 seldepth 10 score mate 1 time 1 nodes 36 nps 36000 hashfull 0 tbhits 24 pv b3b2
info depth 11 seldepth 11 score mate 1 time 1 nodes 39 nps 39000 hashfull 0 tbhits 27 pv b3b2
info depth 12 seldepth 12 score mate 1 time 1 nodes 42 nps 42000 hashfull 0 tbhits 30 pv b3b2
info depth 13 seldepth 13 score mate 1 time 1 nodes 45 nps 45000 hashfull 0 tbhits 33 pv b3b2
info depth 14 seldepth 14 score mate 1 time 1 nodes 48 nps 48000 hashfull 0 tbhits 36 pv b3b2
info depth 15 seldepth 15 score mate 1 time 1 nodes 51 nps 51000 hashfull 0 tbhits 39 pv b3b2
info depth 16 seldepth 16 score mate 1 time 1 nodes 54 nps 54000 hashfull 0 tbhits 42 pv b3b2
info depth 17 seldepth 17 score mate 1 time 1 nodes 57 nps 57000 hashfull 0 tbhits 45 pv b3b2
info depth 18 seldepth 18 score mate 1 time 1 nodes 60 nps 60000 hashfull 0 tbhits 48 pv b3b2
info depth 19 seldepth 19 score mate 1 time 1 nodes 64 nps 64000 hashfull 0 tbhits 52 pv b3b2
info depth 20 seldepth 20 score mate 1 time 1 nodes 68 nps 68000 hashfull 0 tbhits 56 pv b3b2
info depth 21 seldepth 21 score mate 1 time 1 nodes 72 nps 72000 hashfull 0 tbhits 60 pv b3b2
info depth 22 seldepth 22 score mate 1 time 1 nodes 76 nps 76000 hashfull 0 tbhits 64 pv b3b2
info depth 23 seldepth 23 score mate 1 time 1 nodes 80 nps 80000 hashfull 0 tbhits 68 pv b3b2
info depth 24 seldepth 24 score mate 1 time 1 nodes 84 nps 84000 hashfull 0 tbhits 72 pv b3b2
info depth 25 seldepth 25 score mate 1 time 2 nodes 88 nps 44000 hashfull 0 tbhits 76 pv b3b2
info depth 26 seldepth 26 score mate 1 time 2 nodes 92 nps 46000 hashfull 0 tbhits 80 pv b3b2
info depth 27 seldepth 27 score mate 1 time 2 nodes 96 nps 48000 hashfull 0 tbhits 84 pv b3b2
info depth 28 seldepth 28 score mate 1 time 2 nodes 100 nps 50000 hashfull 0 tbhits 88 pv b3b2
info depth 29 seldepth 29 score mate 1 time 2 nodes 104 nps 52000 hashfull 0 tbhits 92 pv b3b2
info depth 30 seldepth 30 score mate 1 time 2 nodes 108 nps 54000 hashfull 0 tbhits 96 pv b3b2
info depth 31 seldepth 31 score mate 1 time 2 nodes 112 nps 56000 hashfull 0 tbhits 100 pv b3b2
info depth 32 seldepth 32 score mate 1 time 2 nodes 116 nps 58000 hashfull 0 tbhits 104 pv b3b2
info depth 33 seldepth 33 score mate 1 time 2 nodes 120 nps 60000 hashfull 0 tbhits 108 pv b3b2
info depth 34 seldepth 34 score mate 1 time 2 nodes 124 nps 62000 hashfull 0 tbhits 112 pv b3b2
info depth 35 seldepth 35 score mate 1 time 2 nodes 128 nps 64000 hashfull 0 tbhits 116 pv b3b2
info depth 36 seldepth 36 score mate 1 time 2 nodes 132 nps 66000 hashfull 0 tbhits 120 pv b3b2
info depth 37 seldepth 37 score mate 1 time 2 nodes 136 nps 68000 hashfull 0 tbhits 124 pv b3b2
info depth 38 seldepth 38 score mate 1 time 2 nodes 140 nps 70000 hashfull 0 tbhits 128 pv b3b2
info depth 39 seldepth 39 score mate 1 time 2 nodes 144 nps 72000 hashfull 0 tbhits 132 pv b3b2
info depth 40 seldepth 40 score mate 1 time 2 nodes 148 nps 74000 hashfull 0 tbhits 136 pv b3b2
info depth 41 seldepth 41 score mate 1 time 2 nodes 152 nps 76000 hashfull 0 tbhits 140 pv b3b2
info depth 42 seldepth 42 score mate 1 time 2 nodes 156 nps 78000 hashfull 0 tbhits 144 pv b3b2
info time 1000 nodes 156 nps 0 hashfull 0 tbhits 144
info time 2001 nodes 156 nps 0 hashfull 0 tbhits 144
info time 3000 nodes 156 nps 0 hashfull 0 tbhits 144
info time 4000 nodes 156 nps 0 hashfull 0 tbhits 144
info time 5001 nodes 156 nps 0 hashfull 0 tbhits 144
info time 6000 nodes 156 nps 0 hashfull 0 tbhits 144
info time 7000 nodes 156 nps 0 hashfull 0 tbhits 144
info time 8001 nodes 156 nps 0 hashfull 0 tbhits 144
stop
info depth 42 seldepth 42 score mate 1 time 8276 nodes 156 nps 18 hashfull 0 tbhits 144 pv b3b2
bestmove b3b2

It does require extra code to stay in analysis mode at the end, waiting for the stop command. So somebody who doesn't care just won't be motivated to write such code.

[expositor]

Commercial engines are already out - the copy protection fell victim to the "nicer on the line" logic.

I've never seen it used, and I didn't realize that anyone besides Shredder actually used this mechanism for copy protection. I'm happy to add it back if that's not the case.

(I'm not entirely sure I know what you mean by "nicer on the line", since I don't think copy protections have anything to do with the how pretty the messages being sent on the wire are.)

By contrast, introducing a new info error message is not useful because the customary way it's done now is using info string, and that works with existing clients as well.

My understanding of HMK is that a legacy client should ignore an `info error` message, so I don't expect it would introduce compatibility issues.

I thought it would be useful for engines to be able to signal in a recognizable way that they received an unexpected message. But perhaps the idea needs adjustment, or perhaps it's a bad idea altogether.

Introducing WDL instead of CP for NN engines sounds nice on the surface - but a lot of clients simply will not be updated, and stuff like that is bad for the ecosystem.

I don't understand how it's any worse for the ecosystem.

Currently, if you want WDL information, you have to have a custom client that either understands new fields in search infos or knows to parse certain info strings.
If WDL information were added to the draft and the draft were adopted, you'd need a newer client to get WDL information and older clients would simply ignore the unrecognized field.
The only difference between the current situation and the hypothetical one is that everyone trying to pass WDL information would have a consistent way to do it, which sounds strictly like an improvement.

The range in the options from 0 to 2^63 - 1 is probably just a spec bug because negative numbers are required e.g. for configuring threshold values such as contempt.

Yes! that's simply incorrect. The author of Frozenight illustrated this in an example:

MinusKelvin `option name Draw value type int min -1000 max 1000`
Kade I had restricted all option integers to be nonnegative because I couldn't think of anything that needed negative integers, but this is an excellent counterexample. I'll widen the range to include negative integers.

That may introduce a breaking change in case of repeated go

It could be a breaking change when using new clients with some legacy engines, but not when using legacy clients with new engines.

Though it might be best to disallow clients from doing this, after all (and to merely make a recommendation for engines).

is the engine expected to keep the last transmitted board position, or will it execute the bestmove on its internal board?

Right now, the draft specifies the last transmitted board position:

A bestmove messaged is well-formed when its body is of the form `bestmove ...` where `...` is an algebraic token that, when interpreted as long algebraic notation, corresponds to a legal move for the side-to-move in the position defined by the last well-formed position message sent by the client in the idle state (or in the starting position if no such position messages have been sent).

Means, existing engines may behave either way.

Out of the engines I've tested, every engine plays from the last transmitted position, with two exceptions:

Last transmitted position Asymptote, Berserk, BlackMarlin, Blunder, Cheng4, Ethereal, Glauraung, Halogen, Inanis, Koivisto, Komodo, Nalwald, Nemorino, Odonata, Rustic, Stockfish, Tantabus, Velvet, Weiawaga, Winter, Zahak
Executes bestmove Lynx
Crashes Princhess

[Ras]

I've never seen it used, and I didn't realize that anyone besides Shredder actually used this mechanism for copy protection.

Even if it's just Shredder - why the move for emulating existing behaviour using different messages? That's what I mean with "nicer on the line". Like in, "cleaning up" the protocol for no reason other than to have it cleaner, and accepting breaking changes in the process.

My understanding of HMK is that a legacy client should ignore an `info error` message, so I don't expect it would introduce compatibility issues.

When using info string error (message), also old clients would do something reasonable, namely displaying the error.

I thought it would be useful for engines to be able to signal in a recognizable way that they received an unexpected message.

I'm using the info string error (message) already in my UCI engine because I check e.g. the position and the legality of the transferred moves. The problem is that capturing the king would crash my engine, and I want it to behave well even with a buggy client. If for no other reason that there's at least some clear indication of what exactly is going wrong. Other examples would be path names for EGTBs. The user expects the engine to use them, but if the engine can't find them, there should be some sort of feedback. Otherwise, especially for position analysis, the engine may not perform at the level the user thinks it should.

I don't understand how it's any worse for the ecosystem.

Old clients with new engines will break the score display and / or adjudication. That's breaking the ecosystem. Don't expect engines or clients to jump on board right away. Instead, rather expect years to see any seizable adoption rate, if at all. That's the scenario that you should plan for.

Also, it's not like engines can simply be updated. Many are not maintained anymore, and CCRL frequently also tests with engine versions other than the most recent, for example in gauntlets.

that everyone trying to pass WDL information would have a consistent way to do it

You could amend that engines based on WDL should also print the score in CP based on some reasonable fixed formula. The main issue here is tons of tournament software that base the adjudication on the score, so that's a critical feature where breaking changes would not be welcome. It's not about what would look "nicer" to the user, it's about whether software breaks in usage.

It could be a breaking change when using new clients with some legacy engines

And that's bad. As it stands, you would need new clients because of the score breakage, but then you would also need old clients because of this breakage. That fragments the ecosystem - and that's what I'm trying to convey. Not breaking the ecosystem is a core strength of UCI. Any improvement that breaks the ecosystem is not an improvement, but a setback.

Though it might be best to disallow clients from doing this, after all (and to merely make a recommendation for engines).

That's a better way. I know it's a frequent critisicm of UCI that oh-so-many bytes are transferred, but it's not like we're on 300 baud acoustic coupler lines for stdin. The overall protocol design is meant to be as stateless as possible.

Out of the engines I've tested, every engine plays from the last transmitted position

Old versions of my engine did execute the move, but at some point, I changed that to the way Stockfish does it. It's not specified either way, but I reasoned that client authors will test with Stockfish for sure and fix their software if it can't run Stockfish. I guess I'm not the only one with this logic so that "if in doubt, do it like Stockfish" is probably somewhat common.

Btw., Demolito crashes if you don't set any position after the initial uci and then issue a go command. I remember its author argued that this was not an engine bug because go without position is not allowed in the first place.

[hgm]

It is a UCI violation, as the specs say the 'go' command should be ignored in this case.

It could be termed 'inconsequential non-compliance', though, as compliant use of the protocol would never reveal it.