TalkChess.com

done (integer, no default)
If you set done=1 during the initial two-second timeout after xboard sends you the "xboard" command, the timeout will end and xboard will not look for any more feature commands before starting normal operation. If you set done=0, the initial timeout is increased to one hour; in this case, you must set done=1 before xboard will enter normal operation.

kasinp wrote: ↑Sat Nov 17, 2018 5:46 pm Imagine a Stockfish version with a particularly nasty bug that causes it to crash after capturing a knight on f4. This scenario will occur infrequently, but will cause SF to lose the game on the spot. A tournament in which this happens in, say 6 games out of 30, will lead to a distorted view of the strength of other engines.

kasinp wrote: ↑Sat Nov 17, 2018 5:46 pm Wins and losses are not random as with the growing number of games they reflect the relative strength of engines with increasing accuracy. OTOH the selection of opponents against which an engine happens to crash is random.

Daniel Shawul wrote: ↑Sat Nov 17, 2018 6:19 pm From what I gather it is not the fault of Scorpio but cutechess-cli.

Apparently cutechess-cli only waits 10 seconds for an engine to load but loading Scorpio neural networks may take upto 30 seconds.
It is not a problem for winboard if an engine takes an hour to initialize because the winboard protocol says this

done (integer, no default)
If you set done=1 during the initial two-second timeout after xboard sends you the "xboard" command, the timeout will end and xboard will not look for any more feature commands before starting normal operation. If you set done=0, the initial timeout is increased to one hour; in this case, you must set done=1 before xboard will enter normal operation.

The xboard protocol provides a way to counter this by doing:
"feature done 0" ... then time-taking operation ... "feature done 1"
So the engine can take upto 1 hour initializing its stuff and there shouldn't be a problem.
I implemented that and expect it to work in every GUI but i guess cutechess-cli just resumes normal operation after waiting only 10 seconds....

I am fine with scorpio getting out of the tournament due to its hangs but it should not be alluded that the cause of the tournament being restarted
is scoprio especially when i did things the rightway and their GUI (cutechess-cli) happens not to implement winboard correctly.

Daniel

	else if (name == "done")
	{
		write("accepted done", Unbuffered);
		m_initTimer->stop();
		
		if (val == "1")
			initialize();
		return;
	}

Branko Radovanovic wrote: ↑Sat Nov 17, 2018 5:09 pm Confusion aside, the very idea of removing the engine with three crashes from competition and discarding all of its games - supposedly in the interest of fairness - is absolutely misguided, because crashes are essentially random events, just as normal wins, losses and draws are, and discarding valid games actually hurts fairness instead of improving it.

Branko Radovanovic wrote: ↑Sat Nov 17, 2018 6:27 pmIf "fairness" of a certain ruleset means "maximizing the prior probability that a stronger engine gets promoted over the weaker one", then removing valid game outcomes will inevitably work against fairness. Using your example, removing 24 actual or potential "valid" outcomes as well as 6 "tainted" outcomes will have the net negative effect in measuring the relative strength of competitors. (That is my hypothesis anyway - I believe it could be proven with a Monte Carlo simulation, for example.)

syzygy wrote: ↑Sat Nov 17, 2018 7:58 pm The inclusion of games of a randomly crashing engine clearly hurts fairness.

Branko Radovanovic wrote: ↑Sat Nov 17, 2018 8:19 pm

syzygy wrote: ↑Sat Nov 17, 2018 7:58 pm The inclusion of games of a randomly crashing engine clearly hurts fairness.

How can one hurt fairness by including non-crashing games of a crashing engine?

On the contrary, it is their removal that hurts fairness. If it's unfair for an engine to receive a point just because its opponent crashed, it is then equally unfair to deduct a fully earned point against such engine from a game in which it didn't crash. If the choices are discard all or discard none, discarding all only seems fairer.

syzygy wrote: ↑Sat Nov 17, 2018 8:32 pm

Branko Radovanovic wrote: ↑Sat Nov 17, 2018 8:19 pm

syzygy wrote: ↑Sat Nov 17, 2018 7:58 pm The inclusion of games of a randomly crashing engine clearly hurts fairness.

How can one hurt fairness by including non-crashing games of a crashing engine?

I now agree that strictly speaking it does not hurt fairness. But it does hurt the tournament because it distorts the results. See my previous post (which I have edited).

syzygy wrote: ↑Sat Nov 17, 2018 8:39 pm Suppose TCEC had a rule that after the tournament is finished, each engine is awarded random bonus points. For all engines the bonus points are drawn from the same random distribution, so this is absolutely and completely fair. But clearly this would just add noise to the tournament results, distorting the tournament's outcome.

A randomly crashing engine is hardly different from such a rule. Compared to a hypothetical tournament in which the engine had not randomly crashed, the crashing engine randomly awards half and full points to some of the other engines.