TalkChess.com

My winboard engine "Nameless" has played many gauntlet matches on arena 3.0 without problem. A recently announced tournament I intend to join will be played on arena 3.5.1 . So I tried a gauntlet match (40/2 repeating) on arena 3.5.1 against an uci engine with epd start positions. It worked smoothly for a few games until I adjudicated a game as draw. Now Nameless did not show PV or make move after the sparring partner played its move for the next game. I investigated and found this likely to happen if user adjudication of any type is done when Nameless is still thinking. The uci engine however is not affected this way. I replaced my engine with Phalanx-XXIV and faced the same problem. Then I tried with crafty_23_7 - problem again. Is this a known issue with arena 3.5.1 for winboard engines or am I missing something?

There's a problem in the Winboard protocol in that the "?" command is only optional during search - unlike UCI's "stop" which is mandatory during search. That's why some Winboard engines, yours maybe included, don't react when the user tries to abort their search.

In Arena, go to menu->engines->tournament, and in the dialogue that opens, click the tab "Start/Quit". The lowest check box, under "Engines", that is "restart engines after each game", and that has to be checked. I recommend an additional second or maybe two as "additional delay".

Ras wrote:There's a problem in the Winboard protocol in that the "?" command is only optional during search - unlike UCI's "stop" which is mandatory during search. That's why some Winboard engines, yours maybe included, don't react when the user tries to abort their search.

That doesn't really seem a reason why you should get this behavior. It should not take very long for an engine to eventually move, even if it does defer all input until it is done searching. (If the engine hasn't crashed.)

BTW, after adjudication the GUI is more likely to send 'force' rather than '?': the latter ('move now') is for getting a move, and after adjudication it doesn't need a move anymore.

Problems should only occur with engines that do not support 'ping', as then the GUI cannot check if the engine has seen the 'force' command, and thus presumably is ready to start a new game. So they might come up with the move that was supposed to be aborted at a time when the GUI has already started the next game, in which that move likely would be illegal. But that would lead to forfeit by illegal move, not the reported behavior.

hgm wrote:That doesn't really seem a reason why you should get this behavior. It should not take very long for an engine to eventually move, even if it does defer all input until it is done searching.

And how long would Arena wait? Depending on the time mode, that could be minutes, which would be unacceptable. The user has adjudicated and expects an immediate reaction from the GUI.

I think there is only one reliable way to deal with that on protocol level. If the engine hasn't sent that it supports the analysis mode, then "?" probably doesn't work during search, and the only way to reliably abort is killing the engine process.

BTW, after adjudication the GUI is more likely to send 'force' rather than '?': the latter ('move now') is for getting a move

Does "force" really interrupt ongoing calculations? I thought it was "?", which is also why engines without "?" don't work for analysis mode. I had never read the Winboard spec so that "force" is supposed to work during search.

Upon re-reading the spec, it appears that this is actually mentioned - but not in the "force" command. Instead, in the explanation for the "ping" command. When explaining the "force" command, there is no mentioning that this should work during search while that is mentioned for "?". No wonder if people don't get this. On the other hand, having two different commands for aborting the search doesn't make sense in the first place.

I've just checked what Arena actually does, and their authors have read the spec the same way as I did. They are using "?". Afterwards, they send the "result" command to the engine, with the result that the user had given. Then Arena starts the next game with "level", "memory", "new", "level" (?!) and "ping".

Problem: some time after that, the Winboard engine (in this case Dorpsgek) comes up with the move from the previous game because "?" is not implemented. This move probably is illegal in the current game or, even worse, it isn't, and the error gets uncovered later. Arena adjudicates the current game as loss for the Winboard engine because it has violated the rules with an illegal move.

Arena starts the next game with the same command sequence as before, but it seems Arena's move scanner has gone out of sync or whatever; the well-known "engine already calculating" comes. As soon as you get this, the only thing that actually works is restarting Arena.

Solution: let Arena just kill the engines and restart them. That's what the "restart engines" checkbox is for. Especially with Winboard engines.

UCI engines usually don't have this problem because "stop" has to be processed during search. The price, of course, is that the engine author can't get away without any kind of threading (or dirty tricks).

The specs from the days of Tim Mann say, in the general section about input/output that preceeds the descriptions of individual commands:

CECP specs (section 6) wrote:Second, on the input side, you are likely to want to poll during your search and stop it if new input has come in. If you implement pondering, you'll need this so that pondering stops when the user makes a move. You should also poll during normal thinking on your move, so that you can implement the "?" (move now) command, and so that you can respond promptly to a "result", "force", or "quit" command if xboard wants to end the game or terminate your engine. Buffered input makes polling more complicated -- when you poll, you must stop your search if there are either characters in the buffer or characters available from the underlying file descriptor.

which clearly describes "force" as one of the commands that should interrupt thinking. In the description of the "?" command it says that it is even permissible to abort thinking on reception of any command, although it suggests that this would then probably be an undecided side effect in the case of the commands post/nopost and easy/hard. (Implying that these preferably should be processed during search, without interrupting the latter. Which isn't especially difficult, as these commands would just set or clear flags used at root level or after the thinking finishes.)

In practice many engines don't do this, however. I am ashamed to admit most of my engines don't do it either. Even those that do ponder and/or analyze.

"?" in theory is the safest way abort a search, as you would be sure to get a move in response (so that you always must ignore the first move that comes). Unfortunately is is also a good candidate for "the most not-implemented command in WB engines". The "force" command during thinking might or might not result in a move, depending on a race condition.

The intended solution to all these problems is "ping". Which is trivial to implement (unlike "?", or processing commands during thinking in general). There really is no excuse for a WB engine not to implement "ping", it is asking for trouble. WinBoard would always use "ping" at game end as a fence to distinguish moves from the engine belonging to the old game from those belonging to the new game.

In short, a WB engine that does not support "ping" should be considered broken. But also, a GUI that would not use "ping" on engines that support it should be considered broken. I am pretty sure Crafty supports "ping".

Restarting the engine process also has its disadvantages. Engines unreceptive to "?" or "force" during thinking likely are also unreceptive to "quit". So although the new engine process would start with a clean slate, the old process might keep running in the background. If it got stuck, and the game was aborted due to time forfeit, it might even keep running forever. Thus consuming CPU, and invalidating the results of the remaining games in the tourney.

hgm wrote:The specs from the days of Tim Mann say, in the general section about input/output that preceeds the descriptions of individual commands

Yeah, but that's implementation details, and that got removed when the spec got cleaned up. On the other hand, you're right that Arena, given its age, and older Winboard engines should be based on the old spec version where that was mentioned, but still in the wrong place.

In practice many engines don't do this, however.

The fact that "?" doesn't have to be processed during search is actually a big Winboard selling point for single threaded midrange engines, like Dorpsgek or NG-Play. It's bad from the user's point of view, as seen in this thread, but it makes programming so much easier.

Peeking into the pipe is what I consider a dirty trick, and it's OS dependent. Going for clean, separate threads opens a whole can of worms with thread synchronisation, out of order execution and all that nasty stuff (or with mysterious bugs after compiler or OS updates, or on certain platforms). That's not an issue for SMP engines that have to deal with all of that anyway, and since all top engines are SMP, this aspect is irrelevant to them.

I am ashamed to admit most of my engines don't do it either. Even those that do ponder and/or analyze.

Then how do you stop analysis?

"?" in theory is the safest way abort a search, as you would be sure to get a move in response (so that you always must ignore the first move that comes). Unfortunately is is also a good candidate for "the most not-implemented command in WB engines".

Yes, and that's because the decision not to implement threading was one important, maybe even the, reason for WB. However, if "?" is not implemented during search, then it's a safe bet that "force" during search is not implemented either, for the very same reason.

The intended solution to all these problems is "ping".

Which Arena doesn't do before sending "new", which is why it has no way to see that the move that finally comes must be from the old game. However, "ping" doesn't really solve the issue that the engine doesn't process the input at all. Even let's say just 10 seconds delay between "move now" and "engine actually moves" are unacceptable in terms of UIs. The other nasty thing is the variance of the delay, which is also unacceptable.

That's why I think WB engines that don't announce support for analysis mode just have to be killed off instead of trying to abort. In this particular case, that's what "restart engines" would do.

In short, a WB engine that does not support "ping" should be considered broken.

NG-Play in its base version doesn't support "ping". I added that in my bugfix version of NG-Play 9.87 because it was trivial enough to do and avoids some trouble.

However, what's broken isn't engines that don't implement this optional command. What's broken is the protocol specification that hasn't made such an important and easy command mandatory.

Restarting the engine process also has its disadvantages. Engines unreceptive to "?" or "force" during thinking likely are also unreceptive to "quit".

Yes, that's what I'd expect. I thought more in terms of killing the process the really hard way, via process ID and OS kill API. Then the engine is started as a new process which has clean state.

Ras wrote:The fact that "?" doesn't have to be processed during search is actually a big Winboard selling point for single threaded midrange engines, like Dorpsgek or NG-Play. It's bad from the user's point of view, as seen in this thread, but it makes programming so much easier.

Agreed. Building a chess engine is difficult enough, and I consider it a good thing that in a first attempt the programmer doesn't have to dig into the problem of peeking pipes or multi-threading to get something that can work. Even if that goes at the expense of some user convenience. Interrupting thinking is something that would only be done on rare occasions anyway, and WinBoard is resistant to it if the engine uses ping (which does not require any special knowledge to implement).

Then how do you stop analysis?

They have a routine MustStop() that is called regularly during search, which peeks the pipe when analyzing or pondering, and reads the clock when thinking. Commands are then only interpreted after the search is aborted.

My newer engines do it properly, reading available input during search, parsing the command, and deciding if it can be processed immediately, or requires a search abort (leaving the command in the input buffer, so that it can be processed again after the search returns.

However, if "?" is not implemented during search, then it's a safe bet that "force" during search is not implemented either, for the very same reason.

Indeed. This is why WinBoard checks when the "force" command has been processed by issuing a "ping" afterwards, and is aware that this can take some time.

Which Arena doesn't do before sending "new", which is why it has no way to see that the move that finally comes must be from the old game.

On engines that do support "ping" this should be considered improper use of the protocol.

However, "ping" doesn't really solve the issue that the engine doesn't process the input at all. Even let's say just 10 seconds delay between "move now" and "engine actually moves" are unacceptable in terms of UIs. The other nasty thing is the variance of the delay, which is also unacceptable.

Well, the UI cannot be blamed for what the engine does. People would just notice that on some engines "move now" is not implemented. Tough luck.

That's why I think WB engines that don't announce support for analysis mode just have to be killed off instead of trying to abort. In this particular case, that's what "restart engines" would do.

The problem is that I never got killing off engines to work reliably under Windows. The Windows API function TerminateProcess() just doesn't seem to work reliably.

However, what's broken isn't engines that don't implement this optional command. What's broken is the protocol specification that hasn't made such an important and easy command mandatory.

"ping" should not be considered an optional command any more than other commands. All commands are intended to make things work well, and not implementing any of them will always cause some loss of functionality. That you can instruct the GUI not to send them, or even have to explicitly request it will send them, is just a mechanism to provide backward compatibility for the benefit of legacy engines. Which then suffer this loss of functionality. But that is considered better than crashing them by sending them commands they do not implement anyway.

Yes, that's what I'd expect. I thought more in terms of killing the process the really hard way, via process ID and OS kill API. Then the engine is started as a new process which has clean state.

Well, WinBoard tries that, for engines that do not terminate fast enough after receiving "quit". (Configurable through an option /delayAfterQuit.) But that often seems to have no effect on the engine at all, and if I do a tournament with buggy engines that tend to hang, it leaves many CPU-eating zombie processes on the machine. There is an external program pv.exe that can kill programs by name of their executable, and that seems to work reliably. WinBoard can be configured to call external commands after each game, and kill rogue engines that way. But it would not work with concurrency, where several instances of the same engine might be at work simultaneously. It is a real problem.

[Edit] I am not sure what you mean when you make a distinction between the new specs and the original specs. Are you referring to http://hgm.nubati.nl/CECP.html ? At the moment this just has the status of a draft for cleaning up the specs, and I cannot exclude that it still needs some revision to fix errors. But in the description of the "ping" command there, it does say "[feature ping=1] should be considered mandatory to have a properly working engine".

hgm wrote:They have a routine MustStop() that is called regularly during search, which peeks the pipe

Ah, the pipe trick.

My newer engines do it properly, reading available input during search

That's how I also do it. Along with mutexes resp. critical sections and stuff.

Which Arena doesn't do before sending "new", which is why it has no way to see that the move that finally comes must be from the old game.

On engines that do support "ping" this should be considered improper use of the protocol.

Yeah I guess. It's also strange that once Arena's move scanner gets out of sync, it usually doesn't recover anymore.

The problem is that I never got killing off engines to work reliably under Windows. The Windows API function TerminateProcess() just doesn't seem to work reliably.

It seems that terminating the process does usually work, except when in kernel mode, but the call is not blocking and returns immediately. The second issue is that killing a process leaves state in shared DLLs in a mess. The third is that it doesn't work on a process tree unless one is using job objects. But that it doesn't do anything at all isn't normal. Proper access rights have been set? However, it looks like a harder problem than I had been thinking.

"ping" should not be considered an optional command any more than other commands. All commands are intended to make things work well, and not implementing any of them will always cause some loss of functionality.

Stable protocol processing is not just functionality. That's why it should have been made mandatory when "feature" was introduced.

Are you referring to http://hgm.nubati.nl/CECP.html ?

I get "server IP not found" for that one. But under http://hgm.nubati.net/newspecs.html , if you do a full text search for "force", you'll find that the move interruption feature of that command is only mentioned in the explanation of the "ping" command. I think the best would be not to mention it at all and always specify "?".

Btw., I've just accidentally been stumbling across that one: "The option NAME can contain spaces, and had must be different for all option features the engine sends." I guess the "had" wasn't meant to be there?

Ras wrote:Stable protocol processing is not just functionality. That's why it should have been made mandatory when "feature" was introduced.

I suppose it would indeed have been wiser to require that that engines that react to 'protover 2' must mandatorily support 'ping' (and 'setboard'?). I can only guess as to why this was not done. One reason would be to make it trivial to convert v1 engines to v2 for the purpose of not having to wait out the 'feature timeout' period when they are launched, by just having them print "feature done=1".

I am not sure if I agree with you that "stable protocol processing is not just functionality". The instability only occurs is a very specific case, playing a multi-game match. From the viewpoint of the user he would lose the match functionality if 'ping' is not implemented.

I get "server IP not found" for that one.

Sorry, my bad. I meant http://hgm.nubati.net/CECP.html ?

But under http://hgm.nubati.net/newspecs.html , if you do a full text search for "force", you'll find that the move interruption feature of that command is only mentioned in the explanation of the "ping" command. I think the best would be not to mention it at all and always specify "?".

Hmm, I had forgotten all about that one. It seems this is an earlier version of the draft for a 'less woolly' CECP specification (modification date reported as Dec 2014), which then evolved into the CECP.html (Jan 2016).

The problem is that such a change would suggest to engine authors they won't have to respond to "force" while existing GUIs might (and in case of WinBoard actually do) send "force" rather than "?". So this breaks backward compatibility for the GUIs.

Btw., I've just accidentally been stumbling across that one: "The option NAME can contain spaces, and had must be different for all option features the engine sends." I guess the "had" wasn't meant to be there?

Indeed, but this paragraph appears to have been dropped in its entirety in the Jan 2016 version.

That newer version does mention that implementing 'ping' is sort of mandatory, and in the description of the 'force' command mention its thinking-interrupting nature.

hgm wrote: ↑Thu Mar 29, 2018 5:59 pm

Ras wrote:There's a problem in the Winboard protocol in that the "?" command is only optional during search - unlike UCI's "stop" which is mandatory during search. That's why some Winboard engines, yours maybe included, don't react when the user tries to abort their search.

That doesn't really seem a reason why you should get this behavior. It should not take very long for an engine to eventually move, even if it does defer all input until it is done searching. (If the engine hasn't crashed.)

BTW, after adjudication the GUI is more likely to send 'force' rather than '?': the latter ('move now') is for getting a move, and after adjudication it doesn't need a move anymore.

Problems should only occur with engines that do not support 'ping', as then the GUI cannot check if the engine has seen the 'force' command, and thus presumably is ready to start a new game. So they might come up with the move that was supposed to be aborted at a time when the GUI has already started the next game, in which that move likely would be illegal. But that would lead to forfeit by illegal move, not the reported behavior.

That's really a pity that Arena Chess has this unsolved bugs, because I think it has the best graphic and options to manage a tournament, having tried also cutechess and BanksiaGUI. I have tried to contact Martin.Blume@playchesswitharena.de but the email doesn't work. Maybe also Tim Mann of Winboard could be easily fix this bugs. It isn't an abandonware because they are developing more recent beta versions for Linux.