PV after stop/readyok

[stevenaaus]

This is breaking my new depth annotation.
Stockfish is sending a PV for a line after being told to stop+isready and sending a readyok.
Is it allowed to do this ?

Code: Select all

Engine: info depth 12 seldepth 2 score cp 40 nodes 1003 nps 501500 time 2 multipv 1 pv b8c6 b1c3 g8f6 h2h3
Engine: info depth 13 seldepth 2 score cp 40 nodes 1770 nps 590000 time 3 multipv 1 pv b8c6 b1c3 g8f6 h2h3
Scid  : stop
Scid  : isready
Engine: readyok
Engine: info depth 14 seldepth 18 score cp 34 nodes 102770 nps 1802982 time 57 multipv 1 pv b8c6 b1c3 g8f6 h2h3 g4f5 e2e4 f5g6 d2d3 c6b4 b3d1 c8d7 d3d4
Scid  : position startpos moves g2g4 d7d5 f1g2 c8g4 c2c4 e7e6 d1b3 d8c8 c4d5 c7c6 d5c6 b8c6
Scid  : go infinite

[Evert]

Hmm… the protocol spec says this:

Code: Select all

* readyok
	This must be sent when the engine has received an "isready" command and has
	processed all input and is ready to accept new commands now.
	It is usually sent after a command that can take some time to be able to wait for the engine,
	but it can be used anytime, even when the engine is searching,
	and must always be answered with "isready".

The phrase "has processed all input and is ready to accept new commands now" is a bit vague. If you interpret that as "has seen the stop command and issued a command to stop the search and is ready for new input", it's fine. However, I would say that the intention is clearly for "stop" to block until the search has actually stopped, before reading (and responding to) other input.

So I think the protocol spec is worded vague enough that it is allowed to do that, but it is against the intent.

[gcramer]

From the UCI protocol:

* isready
... This command must always be answered with "readyok" and can be sent also when the engine is calculating
in which case the engine should also immediately answer with "readyok" without stopping the search.

So the behaviour of stockfish is fine. I suggest to send "isready" after the "bestmove" response, see below.

* stop
stop calculating as soon as possible,
don't forget the "bestmove" ... token when finishing the search

* bestmove
the engine has stopped searching ...

So to be sure that a "stop" is realized by the engine one has to wait for the "bestmove" response, this should definitively be the last response of the preceding search.

I don't think that stockfish is doing worse things, although it might be a bit nasty that stockfish is not stopping immediately. If the stockfish team is changing this behaviour the engine might loose some ELO points (although this must not be bad).

BTW: IMO stockfish is not necessarily the best choice for analysis - stockfish might be the strongest engine in the world in tournaments, but the analysis results are sometimes bad, especially finding a remis combination in positions with imbalanced material is a problem for stockfish.

[Evert]

From the UCI protocol:

* isready
... This command must always be answered with "readyok" and can be sent also when the engine is calculating
in which case the engine should also immediately answer with "readyok" without stopping the search.

So the behaviour of stockfish is fine. I suggest to send "isready" after the "bestmove" response, see below.

Hmm… but on the other hand, in the cited example the engine received a "stop" command first, so technically it's not supposed to be thinking anymore.

It's a bit odd. This

Code: Select all

[isready] is used to synchronize the engine with the GUI. When the GUI has sent a command or
	multiple commands that can take some time to complete,
	this command can be used to wait for the engine to be ready again

suggests that the sequence "stop/isready" is intended to mean for the engine to stop the search, and respond with "readyok" once it's done so (ie, after "bestmove/ponder"). Otherwise the "synchronisation" part of the isready/readyok doesn't really work out.

The protocol spec isn't very clear on how this should work and how the different commands should interact.

(I find UCI to be deceptively messy in practice, just as I find XBoard protocol deceptively clean, but that's a whole different discussion).

[hgm]

As the other posters also mentioned:

The UCI spec is such that 'isready/readyok' cannot be used to detect if an engine is done searching. The search leads its own life next to the command stream, and is controlled by the go/stop/bestmove dialog.

So the only way to verify 'stop' has been processed is to wait for the 'bestmove'. (No need to probe with isready after that, if you did not send any commands after 'stop'.)

[gcramer]

… but on the other hand, in the cited example the engine received a "stop" command first, so technically it's not supposed to be thinking anymore.

The engine must not stop immediately, the UCI protocol says "as soon as possible", this depends on the engines implementation (the engine has to synchronize the threads, for example). The final synchronization has to be done with "bestmove", as H.G. Muller has pointed out.

The protocol spec isn't very clear on how this should work and how the different commands should interact.

I don't agree, I've implemented the UCI protocol in Scidb (GUI), and it works perfectly, even with stress tests, even with stockfish.

[stevenaaus]

Ok - It's not hard to wait for bestmove instead of readyok. Thanks
.... But Stockfish is the only engine i've seen still sending an info after 'readyok'

Finally got my head around fixing the multiple issues, and have
Scid vs PC's depth annotation going great in Linux and OS X.
Fingers crossed for the windows tests. SHould make a release next weekend.

[hgm]

.... But Stockfish is the only engine i've seen still sending an info after 'readyok'

Apparently it is more tardy than other engines in reacting to 'stop'. And a bit more talkative.

And even engines that do obey 'stop' instantaneously can do this occasionally, because there is a race condition where the 'stop' and the PV can both be sent before either of them arrives.

[mcostalba]

As the other posters also mentioned:

The UCI spec is such that 'isready/readyok' cannot be used to detect if an engine is done searching. The search leads its own life next to the command stream, and is controlled by the go/stop/bestmove dialog.

So the only way to verify 'stop' has been processed is to wait for the 'bestmove'. (No need to probe with isready after that, if you did not send any commands after 'stop'.)

Amazingly, in this case I agree.

Actually UCI protocol is correct, it just needs to be read very careful.

1) First of all 'stop':

Code: Select all

* stop
	stop calculating as soon as possible,
	don't forget the "bestmove" and possibly the "ponder" token when finishing the search

So it does not require search is stopped immediately and actually acknowledges that some info is sent even after stop, like bestmove (actually it must be sent after stop, otherwise engine is broken)

2) Then isready/readyok

Code: Select all

* isready
	this is used to synchronize the engine with the GUI. When the GUI has sent a command or
	multiple commands that can take some time to complete,
	this command can be used to wait for the engine to be ready again or
	to ping the engine to find out if it is still alive.
	E.g. this should be sent after setting the path to the tablebases as this can take some time.
	This command is also required once before the engine is asked to do any search
	to wait for the engine to finish initializing.
	This command must always be answered with "readyok" and can be sent also when the engine is calculating
	in which case the engine should also immediately answer with "readyok" without stopping the search.


* readyok
	This must be sent when the engine has received an "isready" command and has
	processed all input and is ready to accept new commands now.
	It is usually sent after a command that can take some time to be able to wait for the engine,
	but it can be used anytime, even when the engine is searching,
	and must always be answered with "isready".

There is no problem to send isready while searching and engine replies with readyok without affecting search in any way.

The use of isready is " to wait for the engine to be ready again or to ping the engine to find out if it is still alive" in which case engine replies with readyok. And this is exactly what SF does: after a stop command it is still alive and also ready to accept new commands. Actually SF can easily accept new commands after a stop has been sent and before bestmove is returned. SF can even accept a new 'go' command at any time after 'stop' and the new 'go' command will start a new search as soon as possible.

The UCI protocol, quite clearly IMO, states the synchronization with the stop shall be bestmove (just read 'stop' description). Instead readyok is used to tell the GUI that it can send new commands at anytime GUI wishes.

[hgm]

Amazingly, in this case I agree.

So there is hope for you yet!