List of bugfree, opensource Linux and MacOSX engines

[hgm]

Meanwhile I compiled the newer version of KingSlayer too, do you mind, if I would rename the internal version number of 0.00?

When this version passes Oliver's test, I will increase the version number in the repository to 1.0, and tag it as an 'official release'.

[OliverBr]

This one hurts: Wyldchess 1.5 crashed after having played over 100.000s games altogether. (Same happened to Murka3 64 once, btw).
So it's a very very rare bug. I hope the author Michael B. can fix this.

Code: Select all

Terminating process of engine WyldChess(34266)
Finished game 18982 (WyldChess vs OliThink 5.9.1): 0-1 {White's connection stalls}

Are you sure that this is really a bug? From the experience with Fairy-Max it seems that CuteChess also gives this error message on time forfeits.

I am absolutely sure.

Since quite some time I have been running tens of millions games on a 32-core server.
I can guarantee, there has never been a "Terminating process" or "connection stalls", unless there is a severe bug in the engine.
It's not possible oversee such incident, because cutechess-cli aborts the complete tournament.

E.g. Stockfish, Glaurung, Weiss 1.x, Fruit, OliThink: It never happened to them, even though each one of those has played many millions games in the cutechess-cli tournaments I was running. (Sparring partners)

Many WB engines do not process input while thinking.

As far as I know cutechess-cli tries to communicate with the engine when time is running out. I think it is supposed to respond.

I don't think it has something to do with being a WB engine:
- OliThink is a WB only engine in every version.
- OliThink5 didn't even implement the ping-command until lately, so it should work without it, too.
- OliPow 2.2.3 is very, very old WB engine which runs out of time sometimes due to bad time management, still there has never been a "connection stall" on it or any other bug-free OliThink version.
- Ah, I forget to mention that Wyldchess was running in uci mode when it happened.

PS: I can though reproduce the issue with a buggy version: OliThink 5.3.2 had a bug that actually occurred every 5000 games or so. Then it happens. It was a clear bug in the engine. see also: http://talkchess.com/forum3/viewtopic.php?f=7&t=74821
PPS: Same happens with Weiss 0.10. So I knew immediately it has a bug. Terje confirmed it. see also http://talkchess.com/forum3/viewtopic.p ... 45#p862745

[hgm]

Well, I am not 100% convinced. When WB engines do not support ping, CuteChess can obviously not use this method. But it can detect the demise of an engine process by other means (EOF on reading, an error on writing to the engine), so it could still produce the error for such engines.

It gave this error for Fairy-Max, and the error went away when I made sure Fairy-Max wouldn't exceed time. I didn't fix any bugs.

When the OS makes an engine unresponsive at a critical time, even UCI engines might not be able to respond to 'isready' in time.

Even if an OS glitch happens only once in 10 million games, and you have 100 engines each playing 100k games, it will happen to one engine that has only played 100k games.

A decisive test would be to modify an engine to intentionally delay every output it gives (well, perhaps except the thinking output) by 1 sec, and see whether CuteChess considers the engine buggy or just slow.

[OliverBr]

Well, I am not 100% convinced. When WB engines do not support ping, CuteChess can obviously not use this method. But it can detect the demise of an engine process by other means (EOF on reading, an error on writing to the engine), so it could still produce the error for such engines.

It gave this error for Fairy-Max, and the error went away when I made sure Fairy-Max wouldn't exceed time. I didn't fix any bugs.

Of course, you have a very good point here.
On the other hand, I tested more than 1000 engines, many of them ran out of time regularly and it did not happen with them.
But then again, some (or many) of the engines that had those "connection stalls" were perhaps just running out of time?!

Yes, I think, this is something worth to be investigated.

A decisive test would be to modify an engine to intentionally delay every output it gives (well, perhaps except the thinking output) by 1 sec, and see whether CuteChess considers the engine buggy or just slow.

Good idea.

[OliverBr]

I pushed a patch to my on-line repository for KingSlayer, which should fix the promotion issue.

It turned out this was 'reproducibly broken'; KingSlayer would never be able to make his own moves after getting a promotion move as input. Apparently I had broken the code when implementing under-promotion, and never sufficiently tested that. The KingSlayer version I have on Windows, which I am always using in the on-line blitz tournaments, was from before this patch (and cannot handle under-promotion).

I had already fixed the same problem earlier, in the KingSlayer derivative that plays Chess with Different Armies; I now ported that solution to the master branch.

Works great now and it's remarkably stronger than fairy-max. A nice addition to the list:

Code: Select all

   # PLAYER             :  RATING  ERROR  POINTS  PLAYED   (%)     W    D     L  D(%)  CFS(%)
   1 KingSlayer 0.00    :     128     15  1350.0    2000  67.5  1243  214   543  10.7     100
   2 OliThink 3.0.7     :       0   ----   650.0    2000  32.5   543  214  1243  10.7     ---

White advantage = 7.99 +/- 7.78
Draw rate (equal opponents) = 11.53 % +/- 0.78

[hgm]

Ummm, I noticed one more thing:

Originally I wrote KingSlayer with PST evaluation only. But later I added many other terms, just to demonstrate how such terms could be calculated. But they are all optional, and engine-defined options can switch them on and off. The latest version in my repository still has them all switched off. The version I have on Windows, which I always run in the monthly on-line blitz tourneys, has a number of those switched on by default:

Code: Select all

// defaults of enables of various eval features, which can be overruled by engine options
int pawnStruct  = 1; // detect and score passers and other pawn types
int shelters    = 1; // judge the quality of the Pawn shelter around the King
int kingSafety  = 0; // evaluate threat to King based on number and kind of attackers to square next to it
int patterns    = 0; // recognize and devaluate trapped pieces, like white N on h8, B on a7, or R boxed in by own K
int mobEval     = 0; // give bonus for having many pseudo-legal moves
int drawishness = 1; // recognize material combinations with poor winning prospects, and reduce the score for those
int recognizers = 1; // recognize some position-dependent draws in end-game (like KPK) that do not quickly resolve

The four terms that are set to 1 are those I am pretty sure of that they have a positive effect. I never took the trouble to actually tune those terms, so it cannot be excluded some of them would actually be detrimental.

[OliverBr]

Well, I am not 100% convinced. When WB engines do not support ping, CuteChess can obviously not use this method. But it can detect the demise of an engine process by other means (EOF on reading, an error on writing to the engine), so it could still produce the error for such engines.

I have additional info about this topic, because I have just reproduced the issue and have the log here:

Code: Select all

561752 >XboardEngine(1247): time 1628
561752 >XboardEngine(1247): f5f6
561752 <XboardEngine(1247):  1.  -123     6  1        g8f8 
561752 <XboardEngine(1247):  2.  -123    29  2        g8f8 f6e6 
561752 <XboardEngine(1247):  3.  -200   152  4        g8f8 f6g5 f8g7 g5h5 
561752 <XboardEngine(1247):  4.  -219   286  6        g8f8 g6g7 f8g8 f6g6 h5h4 g3h4 
561753 <XboardEngine(1247):  5.  -219   612  6        g8f8 g6g7 f8g8 f6g6 h5h4 g3h4 
561753 <XboardEngine(1247):  6.  -325  1740  8        g8f8 g6g7 f8g8 f6g6 b6b5 a4b5 h5h4 g3h4 
561754 <XboardEngine(1247):  7.  -325  4062  8        g8f8 g6g7 f8g8 f6g6 b6b5 a4b5 h5h4 g3h4 
561763 <XboardEngine(1247):  8.  -428 11854 10        g8f8 g6g7 f8g8 f6g6 b6b5 a4b5 h5h4 g3h4 a5a4 b3a4 
561763 <XboardEngine(1247):  9.  -428 29154 10        g8f8 g6g7 f8g8 f6g6 b6b5 a4b5 h5h4 g3h4 a5a4 b3a4 
561763 <XboardEngine(1247): 10.  -320 32675 11        g8f8 g6g7 f8g8 f6g6 h5h4 g3h4 b6b5 a4b5 a5a4 b3b4 a4a3 
561792 <XboardEngine(1247): 11.  -324 158179 12        g8f8 g6g7 f8g8 f6g6 h5h4 g3h4 b6b5 a4b5 a5a4 b3b4 a4a3 g6f6 
577599 >XboardEngine(1247): ?
Terminating process of engine XboardEngine(1247)

You see.. there was enough time, but after 11. it's death end.
After running out of time, cutechess-cli sends an "?" to the engine. Now, when the engine does not answer to this "?", then it is supposed to be dead.

My guess is, when an WB-engine:
1) runs out of time
2) has not implemented the "?" command.
it will be considered "dead".

Solutions, either (or both):
a) Don't run out of time.
b) Implement "?" (similar to "isready" command in UCI, I suppose)

EDIT: I have just checked the sources of this XboardEngine and yes, it has not implemented the "?" command. Later versions of the same engine that have implemented it, they don't have this issue.

EDIT2: To implement b) is not trivial, you need either a "bioskey()" function that peeks the inputs or an extra thread, because it has to be handled during search.

EDIT3: There may be actually a third c) option: Check signal "SIGINT" and interrupt the search on it:

Code: Select all

signal(SIGINT,TermSearch);

[OliverBr]

It looks as the signal "SIGINT" doesn't do anything with cutechess-cli, so this wouldn't help.

To implement "?" actually helps avoiding the issue, but it's not absolutely necessary. It's also ok if the engines responds quickly anything after "?".

The problem occurs exactly when an engine does not check the time itself during search on regular time intervals. It may be still running out of time, but at least it sends something not too late after the timeout. There has to be some time "margin" in cutechess-cli, before it decides "connection stalls".

[OliverBr]

Final evaluation:

It's a bug of the engine, when cutechess-cli announces "connection stalls", because from cutechess-cli point of view:

1) engine runs out of time.
2) engine doesn't respond in time.

cutechess-cli can only think that the engine is stuck.

One thing may debatable: Is the time enough, that cutechess-cli is waiting for something from the engine?

It may be very little time, but on the other hand there are so many engines out there, which do not have the problem, including most of this "bugfree" list. Some engine are going to be kicked of the list after extension tests.

PS:
Off the list will be claudia 0.5.1, fridolin3.1, wyldchess1.5 and olithink413 (yeah, it happened, after 5338 games in the last test )
Onto the list will be probably. arasanx, fairy-max, kingslayer, rubichess

Code: Select all

Terminating process of engine OliThink 4.1.3(56)
Finished game 5240 (OliThink 3.0.7 vs OliThink 4.1.3): 1-0 {Black's connection stalls}

[hgm]

Wll, the CECP specs define the implementation as the '?' command as optional:

?

Move now. If your engine is thinking, it should move immediately; otherwise, the command should be ignored (treated as a no-op). It is permissible for your engine to always ignore the ? command. The only bad consequence is that xboard's Move Now menu command will do nothing.

It is also permissible for your engine to move immediately if it gets any command while thinking, as long as it processes the command right after moving, but it's preferable if you don't do this. For example, xboard may send post, nopost, easy, hard, force, quit, or other commands while the engine is on move.

So this method of detecting whether an engine has crashed should be considered a CuteChess bug. That it doesn't send signals is another CuteChess bug; some engines designed for Linux-only might not wirk then (e.g. never stop pondering).

Of course engines can implement work-around for buggy GUIs. But it seems a bit unreasonable to disqualify them from a list of non-buggy engines just because they do not have such work-arounds. This is like elevating a GUI bug to a standard.

Of course you can argue that you also don't want engines that can forfeit on time on your list. That is OK, but there always is a very small possibility that engines forfeit through no fault of their own. E.g. because the OS swapped them out of memory (because they were idle when not on move) at a critical time where they only had milliseconds left, so that by the time they get their next CPU slice they have already forfeited.

At which move did this Wyldchess 'crash' manifest itself? Was it close to the end of a session? Or were you playing incremental TC?