List of bugfree, opensource Linux and MacOSX engines

[Ras]

So this method of detecting whether an engine has crashed should be considered a CuteChess bug.

I think it's a CECP shortcoming because there's just no reliable way to tell a time forfeit from an engine crash. If a CECP engine doesn't implement "?", it must not also forfeit on time, at least under reasonable time controls. Otherwise, the engine causes too much hassle to be on such a list.

[OliverBr]

Wll, the CECP specs define the implementation as the '?' command as optional:

No, this is not what I am saying.

I am not saying that a engine must implement "?" in order to work, there are engines which do not have the problems and do not have "?".
I am saying, that "?" helps avoiding the issue.

So this method of detecting whether an engine has crashed should be considered a CuteChess bug. That it doesn't send signals is another CuteChess bug; some engines designed for Linux-only might not wirk then (e.g. never stop pondering).

Of course engines can implement work-around for buggy GUIs. But it seems a bit unreasonable to disqualify them from a list of non-buggy engines just because they do not have such work-arounds. This is like elevating a GUI bug to a standard.

I still think this is NOT a cutechess-cli bug!
Cutechess-Cli does anything it can: It sees that the engine is running out of time, the engine is not responding anymore and cutechess-cli *must* assume that the engine is stalling/hanging.

What else can cutechess-cli do?

Moreover, I repeat:
There are engines which played millions of games without any issue with cutechess-cli. Those engine which have troubles are the minority.
Moreover every strong and famous engine, including Stockfish, Ethereal, Glaurung, Fruit, Defenchess, Xiphos, Weiss, Minic, Rubichess etc..etc.. doesn't have this problem.

This is a clear evidence, that the few engines having the issue are buggy, not cutechess-cli.

Of course you can argue that you also don't want engines that can forfeit on time on your list.

No. There are many engines which forfeit on time but do not getting "stalled".
So again this is not what I am saying.
Engines can forfeit on time, but they must stay responsive. Of course.

At which move did this Wyldchess 'crash' manifest itself? Was it close to the end of a session? Or were you playing incremental TC?

I posted the match of Wyldchess here on its GIT page. It was in the endgame.
You are right about that, it's quite problable that it was a timeout and not-responsive issue.

Wyldchess played a big tourney TimeControl 40/12 together with 20 other engines. There were 2 tourneys with 40 engines.
Every other of the 40 engines passed, only Wyldchess had the problem.

[Ras]

Moreover every strong engine, including Stockfish, Ethereal, Glaurung, Fruit, Defenchess, Xiphos, Weiss, Minic etc..etc.. doesn't have this problem.

That's because most (or even all?) of them are using UCI where "isready" has to be answered at all times, even during search. Also, implementation of the "stop" command for ending an ongoing search is mandatory. That's the difference to CECP where treatment of any input during search is optional.

There are UCI engines that don't react to input during search, e.g. Raven, but then it's a clear-cut engine defect.

[OliverBr]

Moreover every strong engine, including Stockfish, Ethereal, Glaurung, Fruit, Defenchess, Xiphos, Weiss, Minic etc..etc.. doesn't have this problem.

That's because most (or even all?) of them are using UCI where "isready" has to be answered at all times, even during search. Also, implementation of the "stop" command for ending an ongoing search is mandatory. That's the difference to CECP where treatment of any input during search is optional.

This may be a good reason. But there CECP Engine who do not have the issue either. OliThink 5 and OliThink 3 are two examples.
mini edit: It may be that they just don't time out...

PS: OlIThink413 implemented "?", anyway I found out last night, that it has the issue, too. I am guessing now that it is real bug in the search and not a timeout.

IMPORTANT EDIT: Wyldchess was running in UCI mode when it happened... And Removing wyldchess is breaking my heart in those times with all those clones. (joking)

[Ras]

But there CECP Engine who do not have the issue either.

Because they don't forfeit on time. It's most dangerous if they don't have hard time aborts, especially if they use root aspiration windows and run into a fail low at root which can take a long time to resolve.

[OliverBr]

But there CECP Engine who do not have the issue either.

Because they don't forfeit on time. It's most dangerous if they don't have hard time aborts, especially if they use root aspiration windows and run into a fail low at root which can take a long time to resolve.

Of course, it's better when they check inputs during search. If not, ponder and analyze are impossible.

[hgm]

I still think this is NOT a cutechess-cli bug!
Cutechess-Cli does anything it can: It sees that the engine is running out of time, the engine is not responding anymore and cutechess-cli *must* assume that the engine is stalling/hanging.

IMO it is not its task to *assume* anything. It should report what it can detect, but it should not make random assumptions on what it cannot possibly know. The only thing that it knows is this point is that the engine does not respond to ?, and the most likely explanation for that is that it doesn't implement the ? command, as most CECP engines don't.

What else can cutechess-cli do?

It could give an accurate result message: "engine flagged, and did not instantly respond to move-now command". The message " connection stalled" should be reserved for broken pipes.

Moreover, I repeat:
There are engines which played millions of games without any issue with cutechess-cli. Those engine which have troubles are the minority.
Moreover every strong engine, including Stockfish, Ethereal, Glaurung, Fruit, Defenchess, Xiphos, Weiss, Minic etc..etc.. doesn't have this problem.

This is a clear evidence, that the few engines having the issue are buggy, not cutechess-cli.

I don't contest your data, but this interpretation of it is not sound. For one, you cannot use behavior of UCI engines to prove the GUIs CECP implementation has no bugs. As this alleged bug is in the handling of time forfeits, even CECP engines that never forfeit would not expose it.

In fact I don't understand how this can even be an issue for discussion at this point. We know what CuteChess does, and we know that this will result in giving the "connection stalled" verdict for CECP engines that just think a fraction of a second too long, and do not implement '?'. If you don't want to disqualify engines from your list because of a simple time forfeit, you should not use the "connection stalled" as a measure.

The UCI equivalent of '?' is 'stop', as UCI searches always return a move. So I suppose CuteChess uses this to probe whether UCI engines are still alive, rather than 'isready'. (To which the 'readyok' response would not mean much, as it is supposed to come during search, and engines with a separate I/O thread would automatically give it even when their search threads have crashed or are in an infinite loop.) But neither 'stop' nor 'isready' are optional command in UCI; engines that would not implement those would not work at all. So it is no surprise that you never see this problem with UCI engines.

I posted the match of Wyldchess here on its GIT page. It was in the endgame.
Wyldchess played a big tourney TimeControl 40/12 together with 20 other engines. There were 2 tourneys with 40 engines.
Every other of the 40 engines passed, only Wyldchess had the problem.

OK, so it happened at move 69, reasonably far away from the 80-move session end, and the engine was UCI, so that the message means there was no instant response to 'stop'. OTOH, you used sessions of 12 sec, and 3/4 of it had already passed, so it could very well be there was only 2, or even 1 sec left on the clock at this point.

The only thing I wanted to point out is that it can sometimes happen on a heavily loaded machine that a program is unresponsive for a second or so, because it got swapped out, and cannot get any CPU slice before it is swapped back in. That happens very infrequently, but it can happen. Some people are struck by lightning. The argument that you know millions of people that have never been struck by lightning is not evidence that there must have been something wrong with the people that were.

[OliverBr]

..But neither 'stop' nor 'isready' ore optional command in UCI; engines that would not implement those would not work at all. So it is no surprise that you never see this problem with UCI engines.

I repeat with pleasure :

The Wyldchess incident occured with Wyldchess running in UCI mode. As did Claudia and Fridolin3.1.

I never said that I never saw this problems with UCI engines. Where did you get this?

[hgm]

Please note our last two postings crossed, (well, I was still editing it), and I think my last paragrapgh already answered your question.

Of course when Wyldchess in this position would reproducibly crash, it would be an entiely different matter. But if one crash in 100M games is not reproducible, I would be inclined to blame the OS rather than the engine.

[OliverBr]

Please note our last two postings crossed, (well, I was still editing it), and I think my last paragrapgh already answered your question.

Of course when Wyldchess in this position would reproducibly crash, it would be an entiely different matter. But if one crash in 100M games is not reproducible, I would be inclined to blame the OS rather than the engine.

Here we have absolutely different opinions IMO, when the big majority of engine doesn't have issues, it's a problem of the minority when it has.

FYI: Wyldchess crashed in game #1325.
Anyway, I will run another tournament for Wyldchess with many more games in order to verify the issue.

PS: I am investigation OliThink 4.1.3 and like always this issue is reproducible and in this case it is obviously no timeout problem:

Code: Select all

240776 >OliThink 4.1.3(35): time 246
240776 >OliThink 4.1.3(35): d6c6
240776 <OliThink 4.1.3(35):  1 -1063      0         4  a7a8 
240776 <OliThink 4.1.3(35):  2 -1081      0        42  a7a8 g6g5 
240777 <OliThink 4.1.3(35):  3 -1101      0       251  a7b8 h1b1 b8c8 g6g5 
240777 <OliThink 4.1.3(35):  4 -32494      0       592  a7b8 h1b1 b8c8 b1b7 c8d8 b7d7 
240777 <OliThink 4.1.3(35):  5 -32494      0       930  a7b8 h1b1 b8c8 b1b7 c8d8 b7d7 
240777 <OliThink 4.1.3(35):  6 -32496      0      1466  a7b8 h1a1 b8c8 a1h8 
240779 <OliThink 4.1.3(35):  7 -32496      0      8213  a7b8 h1a1 b8c8 a1h8 
240797 <OliThink 4.1.3(35):  8 -32496      2     88005  a7b8 h1a1 b8c8 a1h8 
240891 <OliThink 4.1.3(35):  9 -32496     11    425596  a7b8 h1a1 b8c8 a1h8 
240891 <OliThink 4.1.3(35): 70. ... a7b8
240891 <OliThink 4.1.3(35): whisper -32496(9) 425596 nds 3733000 nps 114 ms 13656 evs
240907 >OliThink 4.1.3(35): time 235
240907 >OliThink 4.1.3(35): h1h7
240910 <OliThink 4.1.3(35):  1 -1083      0         3  b8c8 
Terminating process of engine OliThink 4.1.3(35)

White has just calculated the first iteration on move 71 and then nothing. Bug.

[pgn]
[Event "?"]
[Site "?"]
[Date "2020.11.16"]
[Round "501"]
[White "OliThink 4.1.3w"]
[Black "OliThink 4.1.3b"]
[Result "0-1"]

1. Nc3 d5 2. e3 c5 3. Bb5+ Bd7 4. Nf3 Nf6 5. d4 Bxb5 6. Nxb5 e6 7. Bd2 a6 8. Nc3
Nbd7 9. Qe2 Qb6 10. Na4 Qc6 11. Nxc5 Nxc5 12. dxc5 Bxc5 13. Bc3 Ne4 14. Bd4 Bb4+
15. Kd1 f6 16. Ne1 e5 17. f3 exd4 18. exd4 O-O 19. fxe4 dxe4 20. c4 f5 21. Nc2
Bd6 22. Rf1 Rad8 23. Ne3 f4 24. d5 Qb6 25. Nc2 Rde8 26. b3 f3 27. gxf3 exf3
28. Qf2 Qxf2 29. Rxf2 b5 30. a4 bxc4 31. bxc4 Bc5 32. Rf1 Re4 33. Kd2 Rxc4
34. Kd3 Rh4 35. Ne3 Rff4 36. a5 Rh3 37. Nc2 f2+ 38. Ke2 Re4+ 39. Kd2 Kf7
40. Rab1 Kf6 41. Rb7 Rh5 42. Rd7 Ra4 43. d6 Rxh2 44. Rd8 Rh5 45. d7 Rd5+ 46. Kc3
Rxa5 47. Nb4 Bxb4+ 48. Kxb4 Rab5+ 49. Kc4 Rbc5+ 50. Kb3 Rd3+ 51. Kb4 Rc2
52. Rf8+ Ke7 53. R8xf2 Rxf2 54. Rxf2 Rxd7 55. Rh2 h6 56. Re2+ Kf6 57. Kc5 g6
58. Ra2 Ra7 59. Kb6 Ra8 60. Kb7 Re8 61. Rxa6+ Re6 62. Rxe6+ Kxe6 63. Kc7 h5
64. Kd8 h4 65. Ke8 h3 66. Kd8 h2 67. Kc8 Kd6 68. Kb7 h1=Q+ 69. Ka7 Kc6 70. Kb8
Qh7 0-1

[/pgn]