UCI2WB 4.0

[hgm]

This is all handled in the common code of WinBoard and XBoard, so proper treatment of the -niceEngines parameter would be useful. But perhaps I should configure WinBoard such that it runs the engines at the normal priority (-niceEngines 0, which is the default).

As to the killing of engines: I agree that it is in general better to avoid this, but if a engine is hanging, using CPU, what can you do? Perhaps this should also be configurable through an argument to the adapter; I could for instance have WinBoard pass the value of -delayAfterQuit to UCI2WB. This is the time WinBoard would wait for an engine process to exit by itself before it attempts to kill it. Adapters are now exempt from that, but it would be logical when an adapter would take the same timeout period into account as WinBoard would for CECP engines.

[Ras]

This is all handled in the common code of WinBoard and XBoard

Oh, I didn't realise it's a common base, though that makes sense of course.

but of a engine is hanging, using CPU, what can you do?

Shouldn't happen with UCI because isready and stop need to be evaluated during search, and the only clean way to do that is having an input thread. Since this thread is usually blocked on I/O from its stdin, it will receive a dynamic priority raise once it resumes after you sent a UCI command. So it should be responsive even if all threads are computing at full load.

However, there are some UCI engines out there that don't implement the protocol in that they don't have that parallel handling, but then the UCI spec already says that the GUI may kill them at any moment. Another possibility is a rogue implementation that sets the worker threads at high priority to grab maximum computing resources, but developers who go that way have to expect a truckload of issues (like freezing the whole PC).

I could for instance have WinBoard pass the value of -delayAfterQuit to UCI2WB. This is the time WinBoard would wait for an engine process to exit by itself before it attempts to kill it. Adapters are now exempt from that, but it would be logical when an adapter would take the same timeout period into account as WinBoard would for CECP engines.

Looks good. With a workable default value somewhere around one second, that should go unnoticed for most users.

[hgm]

Shouldn't happen with UCI because isready and stop need to be evaluated during search, and the only clean way to do that is having an input thread.

Well, to side-track a bit: this is not how I do it in my engines. Which are all CECP, but I don't see how that really matters. In CECP you also have the equivalent of 'stop' ('move now', or 'force'), and during ponder search have to react to a move (the equivalent of 'ponderhit'), and during analysis you get prompted for periodic updates (which is comparable in handling to 'isready'). I usually do this through polling during the search with PeekNamedPipe(), and I don't see how I would need to do that any different to handle UCI. Just check once every msec (say) for input, if it is ísready' reply immediately, if it is 'stop' set the abortFlag, and on a ponderhit change the search state from ponder to timed, and then just search on.

[Ras]

I usually do this through polling during the search with PeekNamedPipe(), and I don't see how I would need to do that any different to handle UCI.

Well yeah, that will also work, and probably via ioctl() under Linux. You'll need some sort of node based internal timer anyway to avoid checking the clock too often.

[hgm]

Exactly. I usually combine these tests. E.g. in CrazyWa's Search routine I have:

Code: Select all

if((++nodeCount & 0xFFF) == 0) abortFlag |= TimeIsUp(3); // check time limit every 4K nodes

Where TimeIsUp() checks both for timeout and pending input, depending on the kind of search. Something like

Code: Select all

int
TimeIsUp (int mode)
{ // determine if we should stop, depending on time already used, TC mode, time left on clock and from where it is called ('mode')
  if(Input() && DoCommand(1)) return 1; // abort if command waiting that cannot be handled during search
  if(untimedSearch) return 0;           // no timing on analyze or ponder
  int t = ReadClock(0, 0);              // returns time elapsed since search start
  return (t > timeLimit[mode]) ;
}

The routine DoCommand(Boolean searching) is responsible for reading, parsing and executing commands, but it only reads when the input buffer is empty (and clears it after succesfully excuting the command, before returning 0). And before it gets to parsing/executing commands that require a search abort (which in UCI would be anything other than 'isready' or 'ponderhit') it returns 1 if its argument says it was called during search. (Which then leaves the command in the input buffer for when it is called with argument 0 from the main command loop, after the search has unwinded.) In CECP you have to recognize the case where 'usermove' (and the preceding 'time'/'otim') merely signals a ponder hit (which then must be processed during the search); in all other cases it must be treated as an independent command after search abort (ponder miss or analysis).

The low-level routines for doing the actual checks (platform-dependence taken care of by an #ifdef) are:

Code: Select all

#ifdef WIN32 
#    include <windows.h>
     int Input()
     {  // checks for waiting input in pipe
	static int init; static HANDLE inp; DWORD cnt;
	if(!init) inp = GetStdHandle(STD_INPUT_HANDLE);
	if(!PeekNamedPipe(inp, NULL, 0, NULL, &cnt, NULL) &&
	   !GetNumberOfConsoleInputEvents(inp, &cnt)) return 1;
	return cnt;
    }
#else
#    include <sys/time.h>
#    include <sys/times.h>
#    include <sys/ioctl.h>
#    include <unistd.h>
     int GetTickCount() // with thanks to Tord
     {	struct timeval t;
	gettimeofday(&t, NULL);
	return t.tv_sec*1000 + t.tv_usec/1000;
     }
     int Input()
     {
	int cnt;
	if(ioctl(0, FIONREAD, &cnt)) return 1;
	return cnt;
     }
#endif

This is all pretty straightforward, and I like it better than dealing with the complexity of multiple threads, and how to block those when there is nothing to do for them. The main disadvantage is that you poll for input, so that there is a finite response time. But if it is in the millisec range, that isn't really significant. And you can always make the poll frequency dependent on the time left on the clock.

[Ferdy]

Zombie engines.



I stop testing for now.

Just tried Rubichess 1.4 on my python script. In this script engine is run on its own process, while most engines will quit after sending the quit command, Rubichess will not. So I give it 5 sec to do so, If it still does not quit, then I just kill its process. I thought that this engine would be good to test your adaptor's and my script's process-killing capability.

[hgm]

OK, next attempt. I uploaded a new WinBoard-AA.zip package to the same link. (Beware of browser caching; version = 4.9.190618.)

The UCI2WB in this package should now run at normal priority, and kill engines that do not respond to quit. It also protects itself from overflow of the input buffer. (Which could be a potential cause of the UCI2WB crash that occurred.)

The WinBoard in this package refrains from killing (known) adapters. When a separate file for the engine list is specified, it will load and save this file just before and after using Edit Engine List or Save Engine Settings (from the Engine Settings dialog). It will NOT save the list when WinBoard exits, not even if Save Settings on Exit is on. This should prevent changes in the engine list overwriting each other when you have several WinBoard instances running, and close them in the wrong order. Saving on exit is not needed if the only commands that can change the list already have done the saving itself; it could only act to erase what other WinBoard instances have altered in it afterwards.

What remains is the crashes of WinBoard itself. I could not reproduce those, but I have a theory as to what could cause them. When running a tourney WinBoard has to consult the tourney file after each game to see what game it can do next. To prevent several instances running in parallel to accidentally grab the same game, it locks the file wile it is working on it (picking an unplayed game, and markinng it as 'in progress'). The locking action is 'blocking', i.e. if the file is already locked by another WB instance, execution does not proceed until that other instance unlocks it. If this unlocking somehow does not work reliably, it would render WinBoard unresponsive while it waits for it.

To test this hypothesis this new WB version uses a non-blocking call to lock the file, attempting it up to 20 times when it fails (with a 50ms wait in between), and after that gives up: it then throws up an error popup, and drops out of match mode. I have tested whether this works by locking a tournament file WB was working on with a program I wrote for doing that, and indeed the error popup appeared when it tried to pick a new game. The question now is whether this replaces the crashes by this error response.

[Ferdy]

OK, next attempt. I uploaded a new WinBoard-AA.zip package to the same link. (Beware of browser caching; version = 4.9.190618.)

Tried this on 7 winboard instances on 1 tournament.

After 15 games, 5 winboards crashed and 2 are still there but do nothing, tournament has stopped, no more engines and uci2wb and only 2 winboards in task manager.

Here are the 2 winboards, looks like the final move was not executed, it could not finish the games.



trn file

-participants {Deuterium_v2019.2.37.52
Deuterium_v2019.1.36.50_x64_pop
RubiChess-1.4
amoeba_3.0_win64
Bobcat 8.0 x64
Wasp360-x64
Vajolet2_2.7
TogaII401_intelPGO_x64
texel_1.07_x64_pop
SmarThink_v198_x64_SSE3_standalone
senpai_20-x64
arasanx-64-popcnt
baron344-x64
Rodent III 0.277
rofChade
}
-seedBase 356819207
-debug
-debugfile game%d.txt
-tourneyType 2
-tourneyCycles 50
-defaultMatchGames 2
-syncAfterRound false
-syncAfterCycle false
-saveGameFile "C:\WinBoard-AA\WinBoard\WinBoard\t2.pgn"
-loadGameFile "C:\chess\startpos\lowply50startpos.pgn"
-loadGameIndex 0
-loadPositionFile ""
-loadPositionIndex 1
-rewindIndex 0
-usePolyglotBook false
-polyglotBook "default_book.bin"
-bookDepth 12
-bookVariation 50
-discourageOwnBooks false
-defaultHashSize 128
-defaultCacheSizeEGTB 4
-ponderNextMove false
-smpCores 1
-mps 40
-tc 1
-inc -1.00
-results "+--=+-=-+=-+=+=*******"

Later I will test with single instance.

[hgm]

Thanks! So there were no popups complaining about denied access to the tourney file? Then the crashes must have a different reason.

Were the two surviving WinBoard's still responsive? I am a bit puzzled that they could not finish what they are supposed to do, and still stay responsive.

The last thing that was printed in the message field is the PV from the Thinking Output (which is only one move long here). If the engines would have submitted that move, and UCI2WB would have relayed it, WinBoard should have adjudicated the game (checkmate or 50-move draw), resulting in a call to GameEnds (which writes the result in the tourney file, and quits the engines), and then animate the move, display the move + result message, and finally display the new position with a highlight arrow. It obviously has done none of the latter, which suggests it is hanging in GameEnds().

[Ferdy]

Yes there are no popups. Winboard is still responsive, I can press the forward button and the move was executed on the board.

The last part of the debug of those 2 games is the same.

...
moving
moving
moving
moving
moving
exit size-move, size = 33
square size = 33
ExitEvent() during GameEnds(), wait
GameEnds() seems stuck, proceed exiting