Ping, Xboard, and Cutechess-cli

[michiguel]

Cutechess does not send ping after "new", but after several moves have been "forced". For instance this is a fragment of the log caught

: ------> accepted myname
: ------> accepted san
: ------> accepted done
: ------> new
: ------> level 40 0:16 0
: ------> post
: ------> easy
: ------> force
: ------> computer
: ------> name Glaurung
(I think there should be a ping sent here)
: ------> usermove e2e4
: ------> usermove e7e5
: ------> usermove g1f3
: ------> usermove b8c6
: ------> usermove d2d4
: ------> usermove e5d4
: ------> usermove f3d4
: ------> usermove f8c5
: ------> usermove d4c6
: ------> usermove d8f6
: ------> usermove d1d2
: ------> usermove b7c6
: ------> usermove f1d3
: ------> usermove g8e7
: ------> usermove e1g1
: ------> usermove e8g8
: ------> usermove b1c3
: ------> usermove d7d6
: ------> usermove e4e5
: ------> usermove f6e6
: ------> ping 2

Is that desirable?

In UCI, time consuming initialization of the engine is triggered by the command isready as defined by the protocol. This is very good because when you open an engine just to set parameters, you do not want to wait if there is a costly start up. Xboard does not have that explicitly, but it could be mimicked by the command ping and it is implicit by the behavior expected. For instance, in the protocol it says "xboard may send a sequence like "force", "new", "ping". You must not send the pong response until after you have finished executing the "new" command and are ready for the new game to start". So, sending moves to an engine that did not complete the new process does not sound a good idea. Not that "ping 2" was the first ping sent to the engine after all initial transactions of parameters set up.

In the Gaviota version I am about to release, I tried delaying all costly initializations until ping is received to synchronize it with the UCI spirit of isready (if the ping feature was accepted by the GUI, or course). However, that attempt broke xboard compatibility with cutechess. Therefore, I had to remove it (more specifically, I now assume that new internally implies new + ping).

I think it would be safer for cutechess to send ping after new, or at least just before the first usermove. I cannot say it is a bug since the xboard protocol is not really spelling things out, but very close. Xboard protocol should say what commands are guaranteed to be followed by ping before moves on the board are played or the engines is set to think, so the engine knows what kind of guarantees there are to manage the initialization processes more efficiently.

Miguel

[Evert]

In the Gaviota version I am about to release, I tried delaying all costly initializations until ping is received to synchronize it with the UCI spirit of isready (if the ping feature was accepted by the GUI, or course). However, that attempt broke xboard compatibility with cutechess. Therefore, I had to remove it (more specifically, I now assume that new internally implies new + ping).

Hmm… I seem to recall that "feature done=1" is the intended equivalent of "readyok", which I guess makes "protover" the equivalent of "isready".

However, I do agree that in the spirit of "ping" and "pong" cutechess should send ping before the first move.

[hgm]

In UCI, time consuming initialization of the engine is triggered by the command isready as defined by the protocol. This is very good because when you open an engine just to set parameters, you do not want to wait if there is a costly start up.

The reason why UCI oblicatory needs isready after uciok is that it acts as a signal that all options are now set as desired, and in particular that it is now safe to act on any options that were left at their default setting. Otherwise you would run the risk of trying to allocate huge hash by default, only to discover later that the GUI only requests small hash.

Xboard does not have that explicitly, but it could be mimicked by the command ping and it is implicit by the behavior expected. For instance, in the protocol it says "xboard may send a sequence like "force", "new", "ping". You must not send the pong response until after you have finished executing the "new" command and are ready for the new game to start".

More accurate would be to say that pinging can be done always, but after startup is mandatory in UCI, but not in XBoard.

So, sending moves to an engine that did not complete the new process does not sound a good idea. Not that "ping 2" was the first ping sent to the engine after all initial transactions of parameters set up.

As ping is never mandatory in XBoard protocol, there is nothing bad about this.

In the Gaviota version I am about to release, I tried delaying all costly initializations until ping is received to synchronize it with the UCI spirit of isready (if the ping feature was accepted by the GUI, or course). However, that attempt broke xboard compatibility with cutechess. Therefore, I had to remove it (more specifically, I now assume that new internally implies new + ping).

Well, the ping/isready here is not sent for the benefit of the GUI, so it doesn't have to assume anything internally. The reason ping is not mandatory here indeed is that the engine can conclude from the 'new' command that the GUI wants to start a game with the currently set parameters.

That the GUI allows some 'courtesy time' for the engine to implement the settings is an implementation choice that is not guaranteed by the protocol. The GUI could very well start the engine's clock the instant it sends 'new'. Presumably cutechess-cli only starts the clock on receiving the pong after you processed the moves, so it even allows you to process the move before the game starts. It is up to the GUI to decide with which command it wants to synchronize the clocks.

I think it would be safer for cutechess to send ping after new, or at least just before the first usermove. I cannot say it is a bug since the xboard protocol is not really spelling things out, but very close. Xboard protocol should say what commands are guaranteed to be followed by ping before moves on the board are played or the engines is set to think, so the engine knows what kind of guarantees there are to manage the initialization processes more efficiently.

I see no need for this. Proper implementation is to let 'new' trigger the current option settings (as received by the preceding 'memory', 'cores', 'egtpath' instructions) to take effect. That works perfectly, in the absense of any ping. GUIs can probe if you are done with it by sending ping, when they desire to do so.

The only place where XBoard protocol really needs ping is to know if a search that it tried to abort with a force command has terminated with or without producing a move.

[hgm]

Hmm… I seem to recall that "feature done=1" is the intended equivalent of "readyok", which I guess makes "protover" the equivalent of "isready".

This isn't really true. done=1 is the equivalent of 'uciok'. It tells the GUI that the engine has sent all its features. 'protover 2' is the equivalent of 'uci'.

[michiguel]

In UCI, time consuming initialization of the engine is triggered by the command isready as defined by the protocol. This is very good because when you open an engine just to set parameters, you do not want to wait if there is a costly start up.

The reason why UCI oblicatory needs isready after uciok is that it acts as a signal that all options are now set as desired, and in particular that it is now safe to act on any options that were left at their default setting. Otherwise you would run the risk of trying to allocate huge hash by default, only to discover later that the GUI only requests small hash.

Xboard does not have that explicitly, but it could be mimicked by the command ping and it is implicit by the behavior expected. For instance, in the protocol it says "xboard may send a sequence like "force", "new", "ping". You must not send the pong response until after you have finished executing the "new" command and are ready for the new game to start".

More accurate would be to say that pinging can be done always, but after startup is mandatory in UCI, but not in XBoard.

So, sending moves to an engine that did not complete the new process does not sound a good idea. Not that "ping 2" was the first ping sent to the engine after all initial transactions of parameters set up.

As ping is never mandatory in XBoard protocol, there is nothing bad about this.

Ok, then it is a limitation of xboard protocol, not cutechess, unless the initialization is triggered by new (see below).

In the Gaviota version I am about to release, I tried delaying all costly initializations until ping is received to synchronize it with the UCI spirit of isready (if the ping feature was accepted by the GUI, or course). However, that attempt broke xboard compatibility with cutechess. Therefore, I had to remove it (more specifically, I now assume that new internally implies new + ping).

Well, the ping/isready here is not sent for the benefit of the GUI, so it doesn't have to assume anything internally. The reason ping is not mandatory here indeed is that the engine can conclude from the 'new' command that the GUI wants to start a game with the currently set parameters.

That the GUI allows some 'courtesy time' for the engine to implement the settings is an implementation choice that is not guaranteed by the protocol. The GUI could very well start the engine's clock the instant it sends 'new'. Presumably cutechess-cli only starts the clock on receiving the pong after you processed the moves, so it even allows you to process the move before the game starts. It is up to the GUI to decide with which command it wants to synchronize the clocks.

I think it would be safer for cutechess to send ping after new, or at least just before the first usermove. I cannot say it is a bug since the xboard protocol is not really spelling things out, but very close. Xboard protocol should say what commands are guaranteed to be followed by ping before moves on the board are played or the engines is set to think, so the engine knows what kind of guarantees there are to manage the initialization processes more efficiently.

I see no need for this. Proper implementation is to let 'new' trigger the current option settings (as received by the preceding 'memory', 'cores', 'egtpath' instructions) to take effect.

That is what I am actually doing, but then I suggest to spell that out in the documentation. As I read it, it is an interpretation that that is the best point to trigger the initialization, but it is better not to be guessing.

Anyway, cutechess cli sends moves before the first ping, so it has to assume that at one point the engine finished the installation, so it must be assuming that new triggers that.

Miguel

That works perfectly, in the absense of any ping. GUIs can probe if you are done with it by sending ping, when they desire to do so.

The only place where XBoard protocol really needs ping is to know if a search that it tried to abort with a force command has terminated with or without producing a move.

[hgm]

Well, I suppose it really assumes the initialization is triggered by the commands that specify them. E.g. that 'egtpath' would trigger the initialization of EGTs, that 'memory' would make the engine allocate hash memory, etc. In XBoard protocol there really isn't any reason postpone the initialization until 'new'. It is much more straightforward than UCI in this respect. The engine only has to do what it is told to do at the moment it is told to do it (and done doing anything it had to do before). Postponing things to other commands is just something the engine author could decide to do for convenience (whatever he might think that is).

In UCI there is no guarantee you will get certain settings, as they have a default, and the GUI could very well be happy with the default and suppress sending setoption in that case. So it can happen that you receive 'isready' without having received a hash or egtpath option setting. In XBoard protocol the engine does not report a default value for memory use or EGT path. So the GUI must send it memory and EGT commands to define these quantities. If it doesn't send memory, the engine is not guaranteed to have allocated any hash at all. (Some of my engines would not.)

If the GUI knows the commands that is. If it replies with 'rejected memory' or 'rejected egtformats' to the features, then you know for sure the commands will never come, and you could use those 'rejected' commands to allocate a default.

[Sven]

Hmm… I seem to recall that "feature done=1" is the intended equivalent of "readyok", which I guess makes "protover" the equivalent of "isready".

This isn't really true. done=1 is the equivalent of 'uciok'. It tells the GUI that the engine has sent all its features. 'protover 2' is the equivalent of 'uci'.

Formally you are right, but the Chess Engine Communication Protocol also says that the "done=0"/"done=1" pair can be used if the initialization of an engine takes very long, so in fact the intention of "protover 2" regarding the initialization behaviour of an engine is comparable to that of "isready"/"readyok":

Your engine should send one or more feature commands immediately after receiving the "protover" command, since xboard needs to know the values of some features before sending further commands to the engine. Because engines that predate protocol version 2 do not send "feature", xboard uses a timeout mechanism: when it first starts your engine, it sends "xboard" and "protover N", then listens for feature commands for two seconds before sending any other commands. To end this timeout and avoid the wait, set the feature "done=1" at the end of your last feature command. To increase the timeout, if needed, set the feature "done=0" before your first feature command and "done=1" at the end. If needed, it is okay for your engine to set done=0 soon as it starts, even before it receives the xboard and protover commands. This can be useful if your engine takes a long time to initialize itself. It should be harmless even if you are talking to a (version 1) user interface that does not understand the "feature" command, since such interfaces generally ignore commands from the engine that they do not understand.

[hgm]

Formally you are right, but the Chess Engine Communication Protocol also says that the "done=0"/"done=1" pair can be used if the initialization of an engine takes very long, so in fact the intention of "protover 2" regarding the initialization behaviour of an engine is comparable to that of "isready"/"readyok":

Your engine should send one or more feature commands immediately after receiving the "protover" command, since xboard needs to know the values of some features before sending further commands to the engine. Because engines that predate protocol version 2 do not send "feature", xboard uses a timeout mechanism: when it first starts your engine, it sends "xboard" and "protover N", then listens for feature commands for two seconds before sending any other commands. To end this timeout and avoid the wait, set the feature "done=1" at the end of your last feature command. To increase the timeout, if needed, set the feature "done=0" before your first feature command and "done=1" at the end. If needed, it is okay for your engine to set done=0 soon as it starts, even before it receives the xboard and protover commands. This can be useful if your engine takes a long time to initialize itself. It should be harmless even if you are talking to a (version 1) user interface that does not understand the "feature" command, since such interfaces generally ignore commands from the engine that they do not understand.

You can also use 'uciok' to throttle the GUI, in exactly the same way.

I guess this text should better be deleted from the protocol specs, because it seems very bad advice. It would not always work in WinBoard and XBoard, for instance. The described waiting for done=1 is done there only on the first load of the engine. If a new engine process is started for the next game, because the engine emitted feature resue=0, or the user was running with the option -xreuse, there will be no waiting and no timeout. XBoard assumes it already has all the engine features from the previous game. So sending done=0 in the hope it will delay starting your clock is in fact hopeless. It only helps to give you more time to print the features before XBoard starts acting on them.

[Sven]

Formally you are right, but the Chess Engine Communication Protocol also says that the "done=0"/"done=1" pair can be used if the initialization of an engine takes very long, so in fact the intention of "protover 2" regarding the initialization behaviour of an engine is comparable to that of "isready"/"readyok":

Your engine should send one or more feature commands immediately after receiving the "protover" command, since xboard needs to know the values of some features before sending further commands to the engine. Because engines that predate protocol version 2 do not send "feature", xboard uses a timeout mechanism: when it first starts your engine, it sends "xboard" and "protover N", then listens for feature commands for two seconds before sending any other commands. To end this timeout and avoid the wait, set the feature "done=1" at the end of your last feature command. To increase the timeout, if needed, set the feature "done=0" before your first feature command and "done=1" at the end. If needed, it is okay for your engine to set done=0 soon as it starts, even before it receives the xboard and protover commands. This can be useful if your engine takes a long time to initialize itself. It should be harmless even if you are talking to a (version 1) user interface that does not understand the "feature" command, since such interfaces generally ignore commands from the engine that they do not understand.

You can also use 'uciok' to throttle the GUI, in exactly the same way.

I guess this text should better be deleted from the protocol specs, because it seems very bad advice. It would not always work in WinBoard and XBoard, for instance. The described waiting for done=1 is done there only on the first load of the engine. If a new engine process is started for the next game, because the engine emitted feature resue=0, or the user was running with the option -xreuse, there will be no waiting and no timeout. XBoard assumes it already has all the engine features from the previous game. So sending done=0 in the hope it will delay starting your clock is in fact hopeless. It only helps to give you more time to print the features before XBoard starts acting on them.

I'd say most WB engines that have a non-trivial initialization (which is already given if you zero your hash table at startup) rely on that "done=0/done=1" stuff. I would indeed consider it a severe bug that WB behaves differently for the first vs. any subsequent startups of an engine. You cannot offer a "reuse=0" feature in combination with killing and restarting engine processes while denying identical conditions regarding the initialization phase to each incarnation of an engine process.

So you'd better not delete that paragraph. Instead I suggest that you implement the "done=0/done=1" mimic for subsequent engine startups as well, in the way everyone would expect it.

Sven

[Evert]

I'd say most WB engines that have a non-trivial initialization (which is already given if you zero your hash table at startup) rely on that "done=0/done=1" stuff. I would indeed consider it a severe bug that WB behaves differently for the first vs. any subsequent startups of an engine. You cannot offer a "reuse=0" feature in combination with killing and restarting engine processes while denying identical conditions regarding the initialization phase to each incarnation of an engine process.

So you'd better not delete that paragraph. Instead I suggest that you implement the "done=0/done=1" mimic for subsequent engine startups as well, in the way everyone would expect it.

+1

There is no real reason for treating restarts of an engine differently from first starts. In fact, I'd think it'd be easier and safer not to. Perhaps make an exception if you've found out that the engine is protocol v1, otherwise you're just wasting time.

It's also quite bad if there's a published spec that spells out how the GUI and engines should behave, but the GUI that it describes doesn't follow that spec. I'd consider that a bug. Perhaps not a critical one that needs fixing pronto, but fixing it would be better than changing the specs, which would be a bit like declaring darkness the new standard if you can't be bothered to change a dead light bulb.