Problem with arena 3.5.1
Moderators: hgm, Rebel, chrisw
-
- Posts: 100
- Joined: Tue Oct 15, 2013 5:45 pm
Problem with arena 3.5.1
My winboard engine "Nameless" has played many gauntlet matches on arena 3.0 without problem. A recently announced tournament I intend to join will be played on arena 3.5.1 . So I tried a gauntlet match (40/2 repeating) on arena 3.5.1 against an uci engine with epd start positions. It worked smoothly for a few games until I adjudicated a game as draw. Now Nameless did not show PV or make move after the sparring partner played its move for the next game. I investigated and found this likely to happen if user adjudication of any type is done when Nameless is still thinking. The uci engine however is not affected this way. I replaced my engine with Phalanx-XXIV and faced the same problem. Then I tried with crafty_23_7 - problem again. Is this a known issue with arena 3.5.1 for winboard engines or am I missing something?
-
- Posts: 2487
- Joined: Tue Aug 30, 2016 8:19 pm
- Full name: Rasmus Althoff
Re: Problem with arena 3.5.1
There's a problem in the Winboard protocol in that the "?" command is only optional during search - unlike UCI's "stop" which is mandatory during search. That's why some Winboard engines, yours maybe included, don't react when the user tries to abort their search.
In Arena, go to menu->engines->tournament, and in the dialogue that opens, click the tab "Start/Quit". The lowest check box, under "Engines", that is "restart engines after each game", and that has to be checked. I recommend an additional second or maybe two as "additional delay".
In Arena, go to menu->engines->tournament, and in the dialogue that opens, click the tab "Start/Quit". The lowest check box, under "Engines", that is "restart engines after each game", and that has to be checked. I recommend an additional second or maybe two as "additional delay".
-
- Posts: 27790
- Joined: Fri Mar 10, 2006 10:06 am
- Location: Amsterdam
- Full name: H G Muller
Re: Problem with arena 3.5.1
That doesn't really seem a reason why you should get this behavior. It should not take very long for an engine to eventually move, even if it does defer all input until it is done searching. (If the engine hasn't crashed.)Ras wrote:There's a problem in the Winboard protocol in that the "?" command is only optional during search - unlike UCI's "stop" which is mandatory during search. That's why some Winboard engines, yours maybe included, don't react when the user tries to abort their search.
BTW, after adjudication the GUI is more likely to send 'force' rather than '?': the latter ('move now') is for getting a move, and after adjudication it doesn't need a move anymore.
Problems should only occur with engines that do not support 'ping', as then the GUI cannot check if the engine has seen the 'force' command, and thus presumably is ready to start a new game. So they might come up with the move that was supposed to be aborted at a time when the GUI has already started the next game, in which that move likely would be illegal. But that would lead to forfeit by illegal move, not the reported behavior.
-
- Posts: 2487
- Joined: Tue Aug 30, 2016 8:19 pm
- Full name: Rasmus Althoff
Re: Problem with arena 3.5.1
And how long would Arena wait? Depending on the time mode, that could be minutes, which would be unacceptable. The user has adjudicated and expects an immediate reaction from the GUI.hgm wrote:That doesn't really seem a reason why you should get this behavior. It should not take very long for an engine to eventually move, even if it does defer all input until it is done searching.
I think there is only one reliable way to deal with that on protocol level. If the engine hasn't sent that it supports the analysis mode, then "?" probably doesn't work during search, and the only way to reliably abort is killing the engine process.
Does "force" really interrupt ongoing calculations? I thought it was "?", which is also why engines without "?" don't work for analysis mode. I had never read the Winboard spec so that "force" is supposed to work during search.BTW, after adjudication the GUI is more likely to send 'force' rather than '?': the latter ('move now') is for getting a move
Upon re-reading the spec, it appears that this is actually mentioned - but not in the "force" command. Instead, in the explanation for the "ping" command. When explaining the "force" command, there is no mentioning that this should work during search while that is mentioned for "?". No wonder if people don't get this. On the other hand, having two different commands for aborting the search doesn't make sense in the first place.
I've just checked what Arena actually does, and their authors have read the spec the same way as I did. They are using "?". Afterwards, they send the "result" command to the engine, with the result that the user had given. Then Arena starts the next game with "level", "memory", "new", "level" (?!) and "ping".
Problem: some time after that, the Winboard engine (in this case Dorpsgek) comes up with the move from the previous game because "?" is not implemented. This move probably is illegal in the current game or, even worse, it isn't, and the error gets uncovered later. Arena adjudicates the current game as loss for the Winboard engine because it has violated the rules with an illegal move.
Arena starts the next game with the same command sequence as before, but it seems Arena's move scanner has gone out of sync or whatever; the well-known "engine already calculating" comes. As soon as you get this, the only thing that actually works is restarting Arena.
Solution: let Arena just kill the engines and restart them. That's what the "restart engines" checkbox is for. Especially with Winboard engines.
UCI engines usually don't have this problem because "stop" has to be processed during search. The price, of course, is that the engine author can't get away without any kind of threading (or dirty tricks).
-
- Posts: 27790
- Joined: Fri Mar 10, 2006 10:06 am
- Location: Amsterdam
- Full name: H G Muller
Re: Problem with arena 3.5.1
The specs from the days of Tim Mann say, in the general section about input/output that preceeds the descriptions of individual commands:
In practice many engines don't do this, however. I am ashamed to admit most of my engines don't do it either. Even those that do ponder and/or analyze.
"?" in theory is the safest way abort a search, as you would be sure to get a move in response (so that you always must ignore the first move that comes). Unfortunately is is also a good candidate for "the most not-implemented command in WB engines". The "force" command during thinking might or might not result in a move, depending on a race condition.
The intended solution to all these problems is "ping". Which is trivial to implement (unlike "?", or processing commands during thinking in general). There really is no excuse for a WB engine not to implement "ping", it is asking for trouble. WinBoard would always use "ping" at game end as a fence to distinguish moves from the engine belonging to the old game from those belonging to the new game.
In short, a WB engine that does not support "ping" should be considered broken. But also, a GUI that would not use "ping" on engines that support it should be considered broken. I am pretty sure Crafty supports "ping".
Restarting the engine process also has its disadvantages. Engines unreceptive to "?" or "force" during thinking likely are also unreceptive to "quit". So although the new engine process would start with a clean slate, the old process might keep running in the background. If it got stuck, and the game was aborted due to time forfeit, it might even keep running forever. Thus consuming CPU, and invalidating the results of the remaining games in the tourney.
which clearly describes "force" as one of the commands that should interrupt thinking. In the description of the "?" command it says that it is even permissible to abort thinking on reception of any command, although it suggests that this would then probably be an undecided side effect in the case of the commands post/nopost and easy/hard. (Implying that these preferably should be processed during search, without interrupting the latter. Which isn't especially difficult, as these commands would just set or clear flags used at root level or after the thinking finishes.)CECP specs (section 6) wrote:Second, on the input side, you are likely to want to poll during your search and stop it if new input has come in. If you implement pondering, you'll need this so that pondering stops when the user makes a move. You should also poll during normal thinking on your move, so that you can implement the "?" (move now) command, and so that you can respond promptly to a "result", "force", or "quit" command if xboard wants to end the game or terminate your engine. Buffered input makes polling more complicated -- when you poll, you must stop your search if there are either characters in the buffer or characters available from the underlying file descriptor.
In practice many engines don't do this, however. I am ashamed to admit most of my engines don't do it either. Even those that do ponder and/or analyze.
"?" in theory is the safest way abort a search, as you would be sure to get a move in response (so that you always must ignore the first move that comes). Unfortunately is is also a good candidate for "the most not-implemented command in WB engines". The "force" command during thinking might or might not result in a move, depending on a race condition.
The intended solution to all these problems is "ping". Which is trivial to implement (unlike "?", or processing commands during thinking in general). There really is no excuse for a WB engine not to implement "ping", it is asking for trouble. WinBoard would always use "ping" at game end as a fence to distinguish moves from the engine belonging to the old game from those belonging to the new game.
In short, a WB engine that does not support "ping" should be considered broken. But also, a GUI that would not use "ping" on engines that support it should be considered broken. I am pretty sure Crafty supports "ping".
Restarting the engine process also has its disadvantages. Engines unreceptive to "?" or "force" during thinking likely are also unreceptive to "quit". So although the new engine process would start with a clean slate, the old process might keep running in the background. If it got stuck, and the game was aborted due to time forfeit, it might even keep running forever. Thus consuming CPU, and invalidating the results of the remaining games in the tourney.
-
- Posts: 2487
- Joined: Tue Aug 30, 2016 8:19 pm
- Full name: Rasmus Althoff
Re: Problem with arena 3.5.1
Yeah, but that's implementation details, and that got removed when the spec got cleaned up. On the other hand, you're right that Arena, given its age, and older Winboard engines should be based on the old spec version where that was mentioned, but still in the wrong place.hgm wrote:The specs from the days of Tim Mann say, in the general section about input/output that preceeds the descriptions of individual commands
The fact that "?" doesn't have to be processed during search is actually a big Winboard selling point for single threaded midrange engines, like Dorpsgek or NG-Play. It's bad from the user's point of view, as seen in this thread, but it makes programming so much easier.In practice many engines don't do this, however.
Peeking into the pipe is what I consider a dirty trick, and it's OS dependent. Going for clean, separate threads opens a whole can of worms with thread synchronisation, out of order execution and all that nasty stuff (or with mysterious bugs after compiler or OS updates, or on certain platforms). That's not an issue for SMP engines that have to deal with all of that anyway, and since all top engines are SMP, this aspect is irrelevant to them.
Then how do you stop analysis?I am ashamed to admit most of my engines don't do it either. Even those that do ponder and/or analyze.
Yes, and that's because the decision not to implement threading was one important, maybe even the, reason for WB. However, if "?" is not implemented during search, then it's a safe bet that "force" during search is not implemented either, for the very same reason."?" in theory is the safest way abort a search, as you would be sure to get a move in response (so that you always must ignore the first move that comes). Unfortunately is is also a good candidate for "the most not-implemented command in WB engines".
Which Arena doesn't do before sending "new", which is why it has no way to see that the move that finally comes must be from the old game. However, "ping" doesn't really solve the issue that the engine doesn't process the input at all. Even let's say just 10 seconds delay between "move now" and "engine actually moves" are unacceptable in terms of UIs. The other nasty thing is the variance of the delay, which is also unacceptable.The intended solution to all these problems is "ping".
That's why I think WB engines that don't announce support for analysis mode just have to be killed off instead of trying to abort. In this particular case, that's what "restart engines" would do.
NG-Play in its base version doesn't support "ping". I added that in my bugfix version of NG-Play 9.87 because it was trivial enough to do and avoids some trouble.In short, a WB engine that does not support "ping" should be considered broken.
However, what's broken isn't engines that don't implement this optional command. What's broken is the protocol specification that hasn't made such an important and easy command mandatory.
Yes, that's what I'd expect. I thought more in terms of killing the process the really hard way, via process ID and OS kill API. Then the engine is started as a new process which has clean state.Restarting the engine process also has its disadvantages. Engines unreceptive to "?" or "force" during thinking likely are also unreceptive to "quit".
-
- Posts: 27790
- Joined: Fri Mar 10, 2006 10:06 am
- Location: Amsterdam
- Full name: H G Muller
Re: Problem with arena 3.5.1
Agreed. Building a chess engine is difficult enough, and I consider it a good thing that in a first attempt the programmer doesn't have to dig into the problem of peeking pipes or multi-threading to get something that can work. Even if that goes at the expense of some user convenience. Interrupting thinking is something that would only be done on rare occasions anyway, and WinBoard is resistant to it if the engine uses ping (which does not require any special knowledge to implement).Ras wrote:The fact that "?" doesn't have to be processed during search is actually a big Winboard selling point for single threaded midrange engines, like Dorpsgek or NG-Play. It's bad from the user's point of view, as seen in this thread, but it makes programming so much easier.
They have a routine MustStop() that is called regularly during search, which peeks the pipe when analyzing or pondering, and reads the clock when thinking. Commands are then only interpreted after the search is aborted.Then how do you stop analysis?
My newer engines do it properly, reading available input during search, parsing the command, and deciding if it can be processed immediately, or requires a search abort (leaving the command in the input buffer, so that it can be processed again after the search returns.
Indeed. This is why WinBoard checks when the "force" command has been processed by issuing a "ping" afterwards, and is aware that this can take some time.However, if "?" is not implemented during search, then it's a safe bet that "force" during search is not implemented either, for the very same reason.
On engines that do support "ping" this should be considered improper use of the protocol.Which Arena doesn't do before sending "new", which is why it has no way to see that the move that finally comes must be from the old game.
Well, the UI cannot be blamed for what the engine does. People would just notice that on some engines "move now" is not implemented. Tough luck.However, "ping" doesn't really solve the issue that the engine doesn't process the input at all. Even let's say just 10 seconds delay between "move now" and "engine actually moves" are unacceptable in terms of UIs. The other nasty thing is the variance of the delay, which is also unacceptable.
The problem is that I never got killing off engines to work reliably under Windows. The Windows API function TerminateProcess() just doesn't seem to work reliably.That's why I think WB engines that don't announce support for analysis mode just have to be killed off instead of trying to abort. In this particular case, that's what "restart engines" would do.
"ping" should not be considered an optional command any more than other commands. All commands are intended to make things work well, and not implementing any of them will always cause some loss of functionality. That you can instruct the GUI not to send them, or even have to explicitly request it will send them, is just a mechanism to provide backward compatibility for the benefit of legacy engines. Which then suffer this loss of functionality. But that is considered better than crashing them by sending them commands they do not implement anyway.However, what's broken isn't engines that don't implement this optional command. What's broken is the protocol specification that hasn't made such an important and easy command mandatory.
Well, WinBoard tries that, for engines that do not terminate fast enough after receiving "quit". (Configurable through an option /delayAfterQuit.) But that often seems to have no effect on the engine at all, and if I do a tournament with buggy engines that tend to hang, it leaves many CPU-eating zombie processes on the machine. There is an external program pv.exe that can kill programs by name of their executable, and that seems to work reliably. WinBoard can be configured to call external commands after each game, and kill rogue engines that way. But it would not work with concurrency, where several instances of the same engine might be at work simultaneously. It is a real problem.Yes, that's what I'd expect. I thought more in terms of killing the process the really hard way, via process ID and OS kill API. Then the engine is started as a new process which has clean state.
[Edit] I am not sure what you mean when you make a distinction between the new specs and the original specs. Are you referring to http://hgm.nubati.nl/CECP.html ? At the moment this just has the status of a draft for cleaning up the specs, and I cannot exclude that it still needs some revision to fix errors. But in the description of the "ping" command there, it does say "[feature ping=1] should be considered mandatory to have a properly working engine".
-
- Posts: 2487
- Joined: Tue Aug 30, 2016 8:19 pm
- Full name: Rasmus Althoff
Re: Problem with arena 3.5.1
Ah, the pipe trick.hgm wrote:They have a routine MustStop() that is called regularly during search, which peeks the pipe
That's how I also do it. Along with mutexes resp. critical sections and stuff.My newer engines do it properly, reading available input during search
Yeah I guess. It's also strange that once Arena's move scanner gets out of sync, it usually doesn't recover anymore.On engines that do support "ping" this should be considered improper use of the protocol.Which Arena doesn't do before sending "new", which is why it has no way to see that the move that finally comes must be from the old game.
It seems that terminating the process does usually work, except when in kernel mode, but the call is not blocking and returns immediately. The second issue is that killing a process leaves state in shared DLLs in a mess. The third is that it doesn't work on a process tree unless one is using job objects. But that it doesn't do anything at all isn't normal. Proper access rights have been set? However, it looks like a harder problem than I had been thinking.The problem is that I never got killing off engines to work reliably under Windows. The Windows API function TerminateProcess() just doesn't seem to work reliably.
Stable protocol processing is not just functionality. That's why it should have been made mandatory when "feature" was introduced."ping" should not be considered an optional command any more than other commands. All commands are intended to make things work well, and not implementing any of them will always cause some loss of functionality.
I get "server IP not found" for that one. But under http://hgm.nubati.net/newspecs.html , if you do a full text search for "force", you'll find that the move interruption feature of that command is only mentioned in the explanation of the "ping" command. I think the best would be not to mention it at all and always specify "?".Are you referring to http://hgm.nubati.nl/CECP.html ?
Btw., I've just accidentally been stumbling across that one: "The option NAME can contain spaces, and had must be different for all option features the engine sends." I guess the "had" wasn't meant to be there?
-
- Posts: 27790
- Joined: Fri Mar 10, 2006 10:06 am
- Location: Amsterdam
- Full name: H G Muller
Re: Problem with arena 3.5.1
I suppose it would indeed have been wiser to require that that engines that react to 'protover 2' must mandatorily support 'ping' (and 'setboard'?). I can only guess as to why this was not done. One reason would be to make it trivial to convert v1 engines to v2 for the purpose of not having to wait out the 'feature timeout' period when they are launched, by just having them print "feature done=1".Ras wrote:Stable protocol processing is not just functionality. That's why it should have been made mandatory when "feature" was introduced.
I am not sure if I agree with you that "stable protocol processing is not just functionality". The instability only occurs is a very specific case, playing a multi-game match. From the viewpoint of the user he would lose the match functionality if 'ping' is not implemented.
Sorry, my bad. I meant http://hgm.nubati.net/CECP.html ?I get "server IP not found" for that one.
Hmm, I had forgotten all about that one. It seems this is an earlier version of the draft for a 'less woolly' CECP specification (modification date reported as Dec 2014), which then evolved into the CECP.html (Jan 2016).But under http://hgm.nubati.net/newspecs.html , if you do a full text search for "force", you'll find that the move interruption feature of that command is only mentioned in the explanation of the "ping" command. I think the best would be not to mention it at all and always specify "?".
The problem is that such a change would suggest to engine authors they won't have to respond to "force" while existing GUIs might (and in case of WinBoard actually do) send "force" rather than "?". So this breaks backward compatibility for the GUIs.
Indeed, but this paragraph appears to have been dropped in its entirety in the Jan 2016 version.Btw., I've just accidentally been stumbling across that one: "The option NAME can contain spaces, and had must be different for all option features the engine sends." I guess the "had" wasn't meant to be there?
That newer version does mention that implementing 'ping' is sort of mandatory, and in the description of the 'force' command mention its thinking-interrupting nature.
-
- Posts: 1491
- Joined: Sat Feb 06, 2021 8:06 am
- Full name: Alex Morales
Re: Problem with arena 3.5.1
That's really a pity that Arena Chess has this unsolved bugs, because I think it has the best graphic and options to manage a tournament, having tried also cutechess and BanksiaGUI. I have tried to contact Martin.Blume@playchesswitharena.de but the email doesn't work. Maybe also Tim Mann of Winboard could be easily fix this bugs. It isn't an abandonware because they are developing more recent beta versions for Linux.hgm wrote: ↑Thu Mar 29, 2018 5:59 pmThat doesn't really seem a reason why you should get this behavior. It should not take very long for an engine to eventually move, even if it does defer all input until it is done searching. (If the engine hasn't crashed.)Ras wrote:There's a problem in the Winboard protocol in that the "?" command is only optional during search - unlike UCI's "stop" which is mandatory during search. That's why some Winboard engines, yours maybe included, don't react when the user tries to abort their search.
BTW, after adjudication the GUI is more likely to send 'force' rather than '?': the latter ('move now') is for getting a move, and after adjudication it doesn't need a move anymore.
Problems should only occur with engines that do not support 'ping', as then the GUI cannot check if the engine has seen the 'force' command, and thus presumably is ready to start a new game. So they might come up with the move that was supposed to be aborted at a time when the GUI has already started the next game, in which that move likely would be illegal. But that would lead to forfeit by illegal move, not the reported behavior.
Chess engines and dedicated chess computers fan since 1981
Mac mini M1 8GB-256GB, Windows 11 & Ubuntu ARM64.
ProteusSF Dev Forum TROLLS KINDERGARTEN
![Very Happy :D](./images/smilies/icon_biggrin.gif)
ProteusSF Dev Forum TROLLS KINDERGARTEN