Handling UCI protocol and Stockfish / Glaurung behavior

[pgeorges]

Hi,

I think some UCI engine does not handle correctly UCI protocol in some special circumstances, among them Stockfish (and of course some Glaurung versions, I did not check all).

For example with Stockfish 1.6, enter the following commands :

Code: Select all

uci
position fen 7R/p5p1/1p4p1/n1pq2k1/6Q1/P2r3P/6P1/4R2K b - - 6 39
go infinite

It is a mate in 2 but instead of stopping the output at depth 2, SF will output 200 lines of this pattern :

Code: Select all

info depth D
info depth D score mate -2 time 129 nodes 10154 nps 78713 pv g5f6 h8f8 d5f7 g4f4 

where D = [1..100]. This output could be avoided IMHO, making life easier for user interfaces, because parsing and translating this output takes time (the engine bombs the UI very quicly of course because life is pretty easy for it in that case, with a mate in 2 ). Note that if you set SF to use all your CPUs and cores, the UI does not get a lot of CPU power for itself, so it is always good to spare it.

Then just after this, if you enter

Code: Select all

stop

SF replies

Code: Select all

Unknown command: stop

where it should reply, if conforming to the UCI protocol with a "bestmove". See this extract :

* stop
stop calculating as soon as possible,
don't forget the "bestmove" and possibly the "ponder" token when finishing the search

And in the UCI protocol it is said that after a "go infinite" :

search until the "stop" command. Do not exit the search without being told so in this mode!

This is what makes Stockfish feel frozen under Scid (it may affect other interfaces, I guess).

Having said that, I have to say those engines are among the best in the world . Congratulations to the authors !

Pascal Georges

[zullil]

Then just after this, if you enter

Code: Select all

stop

SF replies

Code: Select all

Unknown command: stop

where it should reply, if conforming to the UCI protocol with a "bestmove".
Pascal Georges

You do not receive a bestmove after those 200 lines? I do. And the entire output appears in just a fraction of a second.

As far as I can tell, entering "stop" seems to work correctly for me too.

Code: Select all

./stockfish-1.6 
Stockfish 1.6 64bit. By Tord Romstad, Marco Costalba, Joona Kiiski.
uci
id name Stockfish 1.6 64bit
id author Tord Romstad, Marco Costalba, Joona Kiiski

option name Use Search Log type check default false
option name Search Log Filename type string default SearchLog.txt
option name Book File type string default book.bin
option name Mobility (Middle Game) type spin default 100 min 0 max 200
option name Mobility (Endgame) type spin default 100 min 0 max 200
option name Pawn Structure (Middle Game) type spin default 100 min 0 max 200
option name Pawn Structure (Endgame) type spin default 100 min 0 max 200
option name Passed Pawns (Middle Game) type spin default 100 min 0 max 200
option name Passed Pawns (Endgame) type spin default 100 min 0 max 200
option name Space type spin default 100 min 0 max 200
option name Aggressiveness type spin default 100 min 0 max 200
option name Cowardice type spin default 100 min 0 max 200
option name King Safety Curve type combo default Quadratic var Quadratic var Linear
option name King Safety Coefficient type spin default 40 min 1 max 100
option name King Safety X Intercept type spin default 0 min 0 max 20
option name King Safety Max Slope type spin default 30 min 10 max 100
option name King Safety Max Value type spin default 500 min 100 max 1000
option name Queen Contact Check Bonus type spin default 3 min 0 max 8
option name Queen Check Bonus type spin default 2 min 0 max 4
option name Rook Check Bonus type spin default 1 min 0 max 4
option name Bishop Check Bonus type spin default 1 min 0 max 4
option name Knight Check Bonus type spin default 1 min 0 max 4
option name Discovered Check Bonus type spin default 3 min 0 max 8
option name Mate Threat Bonus type spin default 3 min 0 max 8
option name Check Extension (PV nodes) type spin default 2 min 0 max 2
option name Check Extension (non-PV nodes) type spin default 1 min 0 max 2
option name Single Reply Extension (PV nodes) type spin default 2 min 0 max 2
option name Single Reply Extension (non-PV nodes) type spin default 2 min 0 max 2
option name Mate Threat Extension (PV nodes) type spin default 0 min 0 max 2
option name Mate Threat Extension (non-PV nodes) type spin default 0 min 0 max 2
option name Pawn Push to 7th Extension (PV nodes) type spin default 1 min 0 max 2
option name Pawn Push to 7th Extension (non-PV nodes) type spin default 1 min 0 max 2
option name Passed Pawn Extension (PV nodes) type spin default 1 min 0 max 2
option name Passed Pawn Extension (non-PV nodes) type spin default 0 min 0 max 2
option name Pawn Endgame Extension (PV nodes) type spin default 2 min 0 max 2
option name Pawn Endgame Extension (non-PV nodes) type spin default 2 min 0 max 2
option name Full Depth Moves (PV nodes) type spin default 10 min 1 max 100
option name Full Depth Moves (non-PV nodes) type spin default 3 min 1 max 100
option name Threat Depth type spin default 5 min 0 max 100
option name Randomness type spin default 0 min 0 max 10
option name Minimum Split Depth type spin default 4 min 4 max 7
option name Maximum Number of Threads per Split Point type spin default 5 min 4 max 8
option name Threads type spin default 2 min 1 max 8
option name Hash type spin default 32 min 4 max 4096
option name Clear Hash type button
option name New Game type button
option name Ponder type check default true
option name OwnBook type check default true
option name MultiPV type spin default 1 min 1 max 500
option name UCI_ShowCurrLine type check default false
option name UCI_Chess960 type check default false
option name UCI_AnalyseMode type check default false
uciok
position fen 7R/p5p1/1p4p1/n1pq2k1/6Q1/P2r3P/6P1/4R2K b - - 6 39
go infinite
info depth 1
info depth 1 score cp -626 time 44 nodes 14 nps 318 pv g5f6
info depth 2
info depth 2 score mate -2 time 44 nodes 207 nps 4704 pv g5f6 h8f8 d5f7 g4f4 
info depth 3
info depth 3 score mate -2 time 45 nodes 607 nps 13488 pv g5f6 h8f8 d5f7 g4f4 
info depth 4
info depth 4 score mate -2 time 45 nodes 674 nps 14977 pv g5f6 h8f8 d5f7 g4f4 
info depth 5
info depth 5 score mate -2 time 46 nodes 768 nps 16695 pv g5f6 h8f8 d5f7 g4f4 
info depth 6
info depth 6 score mate -2 time 46 nodes 862 nps 18739 pv g5f6 h8f8 d5f7 g4f4 
info depth 7
info depth 7 score mate -2 time 47 nodes 957 nps 20361 pv g5f6 h8f8 d5f7 g4f4 
info depth 8
info depth 8 score mate -2 time 47 nodes 1052 nps 22382 pv g5f6 h8f8 d5f7 g4f4 
info depth 9
info depth 9 score mate -2 time 48 nodes 1148 nps 23916 pv g5f6 h8f8 d5f7 g4f4 
info depth 10
info depth 10 score mate -2 time 49 nodes 1244 nps 25387 pv g5f6 h8f8 d5f7 g4f4 
info depth 11
info depth 11 score mate -2 time 49 nodes 1341 nps 27367 pv g5f6 h8f8 d5f7 g4f4 
info depth 12
info depth 12 score mate -2 time 50 nodes 1438 nps 28760 pv g5f6 h8f8 d5f7 g4f4 
info depth 13
info depth 13 score mate -2 time 51 nodes 1536 nps 30117 pv g5f6 h8f8 d5f7 g4f4 
info depth 14
info depth 14 score mate -2 time 51 nodes 1634 nps 32039 pv g5f6 h8f8 d5f7 g4f4 
info depth 15
info depth 15 score mate -2 time 52 nodes 1733 nps 33326 pv g5f6 h8f8 d5f7 g4f4 
info depth 16
info depth 16 score mate -2 time 52 nodes 1832 nps 35230 pv g5f6 h8f8 d5f7 g4f4 
info depth 17
info depth 17 score mate -2 time 53 nodes 1932 nps 36452 pv g5f6 h8f8 d5f7 g4f4 
info depth 18
info depth 18 score mate -2 time 53 nodes 2032 nps 38339 pv g5f6 h8f8 d5f7 g4f4 
info depth 19
info depth 19 score mate -2 time 54 nodes 2133 nps 39500 pv g5f6 h8f8 d5f7 g4f4 
info depth 20
info depth 20 score mate -2 time 55 nodes 2234 nps 40618 pv g5f6 h8f8 d5f7 g4f4 
info depth 21
info depth 21 score mate -2 time 55 nodes 2336 nps 42472 pv g5f6 h8f8 d5f7 g4f4 
info depth 22
info depth 22 score mate -2 time 56 nodes 2438 nps 43535 pv g5f6 h8f8 d5f7 g4f4 
info depth 23
info depth 23 score mate -2 time 57 nodes 2541 nps 44578 pv g5f6 h8f8 d5f7 g4f4 
info depth 24
info depth 24 score mate -2 time 57 nodes 2644 nps 46385 pv g5f6 h8f8 d5f7 g4f4 
info depth 25
info depth 25 score mate -2 time 58 nodes 2748 nps 47379 pv g5f6 h8f8 d5f7 g4f4 
info depth 26
info depth 26 score mate -2 time 58 nodes 2852 nps 49172 pv g5f6 h8f8 d5f7 g4f4 
info depth 27
info depth 27 score mate -2 time 59 nodes 2957 nps 50118 pv g5f6 h8f8 d5f7 g4f4 
info depth 28
info depth 28 score mate -2 time 60 nodes 3062 nps 51033 pv g5f6 h8f8 d5f7 g4f4 
info depth 29
info depth 29 score mate -2 time 60 nodes 3168 nps 52800 pv g5f6 h8f8 d5f7 g4f4 
info depth 30
info depth 30 score mate -2 time 61 nodes 3274 nps 53672 pv g5f6 h8f8 d5f7 g4f4 
info depth 31
info depth 31 score mate -2 time 62 nodes 3381 nps 54532 pv g5f6 h8f8 d5f7 g4f4 
info depth 32
info depth 32 score mate -2 time 62 nodes 3488 nps 56258 pv g5f6 h8f8 d5f7 g4f4 
info depth 33
info depth 33 score mate -2 time 63 nodes 3596 nps 57079 pv g5f6 h8f8 d5f7 g4f4 
info depth 34
info depth 34 score mate -2 time 64 nodes 3704 nps 57875 pv g5f6 h8f8 d5f7 g4f4 
info depth 35
info depth 35 score mate -2 time 64 nodes 3813 nps 59578 pv g5f6 h8f8 d5f7 g4f4 
info depth 36
info depth 36 score mate -2 time 65 nodes 3922 nps 60338 pv g5f6 h8f8 d5f7 g4f4 
info depth 37
info depth 37 score mate -2 time 66 nodes 4032 nps 61090 pv g5f6 h8f8 d5f7 g4f4 
info depth 38
info depth 38 score mate -2 time 66 nodes 4142 nps 62757 pv g5f6 h8f8 d5f7 g4f4 
info depth 39
info depth 39 score mate -2 time 67 nodes 4253 nps 63477 pv g5f6 h8f8 d5f7 g4f4 
info depth 40
info depth 40 score mate -2 time 68 nodes 4364 nps 64176 pv g5f6 h8f8 d5f7 g4f4 
info depth 41
info depth 41 score mate -2 time 69 nodes 4476 nps 64869 pv g5f6 h8f8 d5f7 g4f4 
info depth 42
info depth 42 score mate -2 time 70 nodes 4588 nps 65542 pv g5f6 h8f8 d5f7 g4f4 
info depth 43
info depth 43 score mate -2 time 70 nodes 4701 nps 67157 pv g5f6 h8f8 d5f7 g4f4 
info depth 44
info depth 44 score mate -2 time 71 nodes 4814 nps 67802 pv g5f6 h8f8 d5f7 g4f4 
info depth 45
info depth 45 score mate -2 time 72 nodes 4928 nps 68444 pv g5f6 h8f8 d5f7 g4f4 
info depth 46
info depth 46 score mate -2 time 73 nodes 5042 nps 69068 pv g5f6 h8f8 d5f7 g4f4 
info depth 47
info depth 47 score mate -2 time 74 nodes 5157 nps 69689 pv g5f6 h8f8 d5f7 g4f4 
info depth 48
info depth 48 score mate -2 time 74 nodes 5272 nps 71243 pv g5f6 h8f8 d5f7 g4f4 
info depth 49
info depth 49 score mate -2 time 75 nodes 5388 nps 71840 pv g5f6 h8f8 d5f7 g4f4 
info depth 50
info depth 50 score mate -2 time 76 nodes 5504 nps 72421 pv g5f6 h8f8 d5f7 g4f4 
info depth 51
info depth 51 score mate -2 time 77 nodes 5621 nps 73000 pv g5f6 h8f8 d5f7 g4f4 
info depth 52
info depth 52 score mate -2 time 77 nodes 5738 nps 74519 pv g5f6 h8f8 d5f7 g4f4 
info depth 53
info depth 53 score mate -2 time 87 nodes 5856 nps 67310 pv g5f6 h8f8 d5f7 g4f4 
info depth 54
info depth 54 score mate -2 time 88 nodes 5974 nps 67886 pv g5f6 h8f8 d5f7 g4f4 
info depth 55
info depth 55 score mate -2 time 88 nodes 6093 nps 69238 pv g5f6 h8f8 d5f7 g4f4 
info depth 56
info depth 56 score mate -2 time 89 nodes 6212 nps 69797 pv g5f6 h8f8 d5f7 g4f4 
info depth 57
info depth 57 score mate -2 time 90 nodes 6332 nps 70355 pv g5f6 h8f8 d5f7 g4f4 
info depth 58
info depth 58 score mate -2 time 90 nodes 6452 nps 71688 pv g5f6 h8f8 d5f7 g4f4 
info depth 59
info depth 59 score mate -2 time 91 nodes 6573 nps 72230 pv g5f6 h8f8 d5f7 g4f4 
info depth 60
info depth 60 score mate -2 time 92 nodes 6694 nps 72760 pv g5f6 h8f8 d5f7 g4f4 
info depth 61
info depth 61 score mate -2 time 92 nodes 6816 nps 74086 pv g5f6 h8f8 d5f7 g4f4 
info depth 62
info depth 62 score mate -2 time 93 nodes 6938 nps 74602 pv g5f6 h8f8 d5f7 g4f4 
info depth 63
info depth 63 score mate -2 time 94 nodes 7061 nps 75117 pv g5f6 h8f8 d5f7 g4f4 
info depth 64
info depth 64 score mate -2 time 95 nodes 7184 nps 75621 pv g5f6 h8f8 d5f7 g4f4 
info depth 65
info depth 65 score mate -2 time 96 nodes 7308 nps 76125 pv g5f6 h8f8 d5f7 g4f4 
info depth 66
info depth 66 score mate -2 time 96 nodes 7432 nps 77416 pv g5f6 h8f8 d5f7 g4f4 
info depth 67
info depth 67 score mate -2 time 97 nodes 7557 nps 77907 pv g5f6 h8f8 d5f7 g4f4 
info depth 68
info depth 68 score mate -2 time 97 nodes 7682 nps 79195 pv g5f6 h8f8 d5f7 g4f4 
info depth 69
info depth 69 score mate -2 time 98 nodes 7808 nps 79673 pv g5f6 h8f8 d5f7 g4f4 
info depth 70
info depth 70 score mate -2 time 99 nodes 7934 nps 80141 pv g5f6 h8f8 d5f7 g4f4 
info depth 71
info depth 71 score mate -2 time 100 nodes 8061 nps 80610 pv g5f6 h8f8 d5f7 g4f4 
info depth 72
info depth 72 score mate -2 time 101 nodes 8188 nps 81069 pv g5f6 h8f8 d5f7 g4f4 
info depth 73
info depth 73 score mate -2 time 101 nodes 8316 nps 82336 pv g5f6 h8f8 d5f7 g4f4 
info depth 74
info depth 74 score mate -2 time 102 nodes 8444 nps 82784 pv g5f6 h8f8 d5f7 g4f4 
info depth 75
info depth 75 score mate -2 time 102 nodes 8573 nps 84049 pv g5f6 h8f8 d5f7 g4f4 
info depth 76
info depth 76 score mate -2 time 103 nodes 8702 nps 84485 pv g5f6 h8f8 d5f7 g4f4 
info depth 77
info depth 77 score mate -2 time 104 nodes 8832 nps 84923 pv g5f6 h8f8 d5f7 g4f4 
info depth 78
info depth 78 score mate -2 time 104 nodes 8962 nps 86173 pv g5f6 h8f8 d5f7 g4f4 
info depth 79
info depth 79 score mate -2 time 105 nodes 9093 nps 86600 pv g5f6 h8f8 d5f7 g4f4 
info depth 80
info depth 80 score mate -2 time 106 nodes 9224 nps 87018 pv g5f6 h8f8 d5f7 g4f4 
info depth 81
info depth 81 score mate -2 time 107 nodes 9356 nps 87439 pv g5f6 h8f8 d5f7 g4f4 
info depth 82
info depth 82 score mate -2 time 107 nodes 9488 nps 88672 pv g5f6 h8f8 d5f7 g4f4 
info depth 83
info depth 83 score mate -2 time 107 nodes 9621 nps 89915 pv g5f6 h8f8 d5f7 g4f4 
info depth 84
info depth 84 score mate -2 time 108 nodes 9754 nps 90314 pv g5f6 h8f8 d5f7 g4f4 
info depth 85
info depth 85 score mate -2 time 108 nodes 9888 nps 91555 pv g5f6 h8f8 d5f7 g4f4 
info depth 86
info depth 86 score mate -2 time 109 nodes 10022 nps 91944 pv g5f6 h8f8 d5f7 g4f4 
info depth 87
info depth 87 score mate -2 time 109 nodes 10157 nps 93183 pv g5f6 h8f8 d5f7 g4f4 
info depth 88
info depth 88 score mate -2 time 110 nodes 10292 nps 93563 pv g5f6 h8f8 d5f7 g4f4 
info depth 89
info depth 89 score mate -2 time 110 nodes 10428 nps 94800 pv g5f6 h8f8 d5f7 g4f4 
info depth 90
info depth 90 score mate -2 time 110 nodes 10564 nps 96036 pv g5f6 h8f8 d5f7 g4f4 
info depth 91
info depth 91 score mate -2 time 111 nodes 10701 nps 96405 pv g5f6 h8f8 d5f7 g4f4 
info depth 92
info depth 92 score mate -2 time 112 nodes 10838 nps 96767 pv g5f6 h8f8 d5f7 g4f4 
info depth 93
info depth 93 score mate -2 time 112 nodes 10976 nps 98000 pv g5f6 h8f8 d5f7 g4f4 
info depth 94
info depth 94 score mate -2 time 113 nodes 11114 nps 98353 pv g5f6 h8f8 d5f7 g4f4 
info depth 95
info depth 95 score mate -2 time 113 nodes 11253 nps 99584 pv g5f6 h8f8 d5f7 g4f4 
info depth 96
info depth 96 score mate -2 time 114 nodes 11392 nps 99929 pv g5f6 h8f8 d5f7 g4f4 
info depth 97
info depth 97 score mate -2 time 114 nodes 11532 nps 101157 pv g5f6 h8f8 d5f7 g4f4 
info depth 98
info depth 98 score mate -2 time 115 nodes 11672 nps 101495 pv g5f6 h8f8 d5f7 g4f4 
info depth 99
info depth 99 score mate -2 time 115 nodes 11813 nps 102721 pv g5f6 h8f8 d5f7 g4f4 
info depth 100
info depth 100 score mate -2 time 115 nodes 11954 nps 103947 pv g5f6 h8f8 d5f7 g4f4 
info nodes 11954 nps 103947 time 115 hashfull 0
bestmove g5f6 ponder h8f8

[Gian-Carlo Pascutto]

You do not receive a bestmove after those 200 lines? I do.

That's the bug - the engine cannot return bestmove by itself while in infinite thinking mode.

The Shredder GUI issues a "buggy engine" warning if you do this.

[mcostalba]

Then just after this, if you enter

Code: Select all

stop

SF replies

Code: Select all

Unknown command: stop

Because has already stopped.

"stop" and "ponderhit" are handled while searching, in poll() function in search.cpp, while other commands that are handled during waiting for user input are handled in uci.cpp in function handle_command()

So answer is "unknown command" because you send this command once engine has already terminated the search and is already stopped (and as already sent a best move BTW).

[Matthias Gemuh]

Then just after this, if you enter

Code: Select all

stop

SF replies

Code: Select all

Unknown command: stop

Because has already stopped.

"stop" and "ponderhit" are handled while searching, in poll() function in search.cpp, while other commands that are handled during waiting for user input are handled in uci.cpp in function handle_command()

So answer is "unknown command" because you send this command once engine has already terminated the search and is already stopped (and as already sent a best move BTW).

The flaw is in the UCI specification.
Why should one search infinitely long for a mate in 2 ?
Only recently did I figure out how to search endlessly long for that which I already have.

Matthias.

[zullil]

You do not receive a bestmove after those 200 lines? I do.

That's the bug - the engine cannot return bestmove by itself while in infinite thinking mode.

The Shredder GUI issues a "buggy engine" warning if you do this.

Oh! I did not read Pascal's post carefully. So "go infinite" should only be terminated by "stop"?

[pgeorges]

You do not receive a bestmove after those 200 lines? I do. And the entire output appears in just a fraction of a second.

Indeed, you get a "bestmove" output but this should be sent after a "stop" command, if you conform to UCI protocol (see the extracts I made above). There is a sequence to follow if you conform to UCI protocol : the engine when asked to enter in analyze mode should not leave that state. Of course the engine can issue a bestmove before the "stop" command (this is even required if you read the description of "bestmove"), but my understanding is that you must repeat it after "stop" because it is what the protocol asks for.

Pascal Georges

[pgeorges]

You do not receive a bestmove after those 200 lines? I do.

That's the bug - the engine cannot return bestmove by itself while in infinite thinking mode.

The Shredder GUI issues a "buggy engine" warning if you do this.

In fact the engine can output a bestmove by itself, see this extract from the protocol :

* bestmove <move1> [ ponder <move2> ]
the engine has stopped searching and found the move <move> best in this position.
the engine can send the move it likes to ponder on. The engine must not start pondering automatically.
this command must always be sent if the engine stops searching, also in pondering mode if there is a
"stop" command, so for every "go" command a "bestmove" command is needed!
Directly before that the engine should send a final "info" command with the final search information,
the the GUI has the complete statistics about the last search.

So if the engine has completed its search, it must send a "bestmove" but also after receiving the "stop" command.

Pascal Georges

[Tord Romstad]

Then just after this, if you enter

Code: Select all

stop

SF replies

Code: Select all

Unknown command: stop

Because has already stopped.

"stop" and "ponderhit" are handled while searching, in poll() function in search.cpp, while other commands that are handled during waiting for user input are handled in uci.cpp in function handle_command()

So answer is "unknown command" because you send this command once engine has already terminated the search and is already stopped (and as already sent a best move BTW).

True, but as Pascal and Gian-Carlo points out, this is a bug. The engine shouldn't send a "bestmove" before receiving a "stop" command when doing an infinite search.

I have no time to check right now, but I think this can be fixed by changing a single line of code. In the id_loop() function, change this code fragment:

Code: Select all

    // If we are pondering, we shouldn't print the best move before we
    // are told to do so
    if (PonderSearch)
      wait_for_stop_or_ponderhit();

To this:

Code: Select all

    // If we are pondering, we shouldn't print the best move before we
    // are told to do so
    if (PonderSearch || InfiniteSearch)
      wait_for_stop_or_ponderhit();

As for outputting 200 lines despite finding a mate quickly, I don't consider this a serious problem, and I don't want to do anything about it. When playing games, Stockfish makes a move quickly when it has found a forced mate. In analysis mode, however, it keeps searching, because it is possible that a faster mate will be found at some later iteration. Of course this won't ever happen for a mate in 2, but writing a special case for a mate in 2 is a little too ugly for my taste.

Thanks for the bug report, Pascal!

[pgeorges]

Then just after this, if you enter

Code: Select all

stop

SF replies

Code: Select all

Unknown command: stop

Because has already stopped.

"stop" and "ponderhit" are handled while searching, in poll() function in search.cpp, while other commands that are handled during waiting for user input are handled in uci.cpp in function handle_command()

So answer is "unknown command" because you send this command once engine has already terminated the search and is already stopped (and as already sent a best move BTW).

Marco, I also found this in SF code, but I think the behavior is still not correct, and does not stick completely to UCI protocol (see my other posts). Of course assuming I got the UCI protocol

Pascal Georges