List of bugfree, opensource Linux and MacOSX engines

[Ras]

Unfortunately both sources, either from your link or from git (https://github.com/giuliano-ippoliti/GiuChess) make quite a number of illegal moves

That's no wonder because it wrongly announces setboard support with "setboard=1", but doesn't actually evaluate a "setboard" command, and it doesn't even support "edit" either. The only way it can support a full game is via consecutive moves, and that's not a complete CECP implementation.

[OliverBr]

Note that Monchester is nearly insta-mover, using constant search depth 4 (you seem to have mentioned instant-movement as negative about e.g. MSCP but I think that itself is not unqualifying). If you happen to find on any bugs in Monchester, I would be glad to hear about them and squash them.

Manchester is a most interesting engine and I hadn't tested it before: It compiles easily on both systems and is very stable. But there is still a issue:
It cannot handle fast time controls. It loses many times on a 40/25 time control (25 seconds for 40 moves). It would make it into the second list (candidates with one minor flaw).
As it is CECP protocol, just read the "time 1023" command from the GUI and interrupt the search before this time has passed. I think this is a very important feature for your engine.

Code: Select all

Finished game 2 (OliThink 4.1.3 vs Monchester 1.0): 1-0 {Black loses on time}
Score of Monchester 1.0 vs OliThink 4.1.3: 0 - 10 - 0  [0.000] 10
Started game 42 of 2000 (OliThink 4.1.3 vs Monchester 1.0)
Finished game 6 (OliThink 4.1.3 vs Monchester 1.0): 1-0 {Black loses on time}
Score of Monchester 1.0 vs OliThink 4.1.3: 0 - 11 - 0  [0.000] 11
Started game 43 of 2000 (Monchester 1.0 vs OliThink 4.1.3)
Finished game 13 (Monchester 1.0 vs OliThink 4.1.3): 0-1 {Black mates}
Score of Monchester 1.0 vs OliThink 4.1.3: 0 - 12 - 0  [0.000] 12
Started game 44 of 2000 (OliThink 4.1.3 vs Monchester 1.0)
Finished game 18 (OliThink 4.1.3 vs Monchester 1.0): 1-0 {Black loses on time}
Score of Monchester 1.0 vs OliThink 4.1.3: 0 - 13 - 0  [0.000] 13
Started game 45 of 2000 (Monchester 1.0 vs OliThink 4.1.3)
Finished game 20 (OliThink 4.1.3 vs Monchester 1.0): 1-0 {Black loses on time}
Score of Monchester 1.0 vs OliThink 4.1.3: 0 - 14 - 0  [0.000] 14

[lucasart]

For completition I repost the list ordered by strength:

Code: Select all

stockfish       uci     +920    11      https://github.com/official-stockfish/Stockfish
ethereal        uci     +700    12.55   https://github.com/AndyGrant/Ethereal
xiphos          uci     +620    0.6.1   https://github.com/milostatarevic/xiphos
defenchess      uci     +600    2.3     https://github.com/cetincan0/Defenchess
laser           uci     +500    1.8     https://github.com/jeffreyan11/laser-chess-engine
demolito        uci     +450    .       https://github.com/lucasart/Demolito
texel           uci     +330    1.08    https://github.com/peterosterlund2/texel                (cmake)
winter          uci     +300    0.8.16  https://github.com/rosenthj/Winter
protector       xb+uci  +280    1.9.0   https://github.com/MichaelB7/Protector
weiss           uci     +250    1.1     https://github.com/TerjeKir/weiss
monolith        uci     +220    2       https://github.com/cimarronOST/Monolith
rodentII        uci     +80     0.9.68  https://github.com/nescitus/Rodent_II
senpai          uci     +40     2.0     https://github.com/B4dT0bi/senpai
delocto         uci     +10     0.6     https://github.com/moterink/Delocto
olithink5       xboard          5.9.0   https://github.com/olithink/OliThink            (/oliThink4 4.1.3: -700, /oliThink3 3.0.7: -950)
halogen         uci     -15     8       https://github.com/KierenP/Halogen
glaurung        uci     -40     2.2     https://github.com/phenri/glaurung
togaII          uci     -70     1.2.1a  https://github.com/zwegner/toga-mp
k2              xb+uci  -80     0.96    https://github.com/serg-meus/k2
scorpio         xb+uci  -130    3.0.9   https://github.com/dshawul/Scorpio
greko           xb+uci  -150    2020.03 http://greko.su/index_en.html
stash           uci     -160    21.1    https://gitlab.com/mhouppin/stash-bot
wyldchess       uci     -160    1.5     https://github.com/Mk-Chan/WyldChess
fruit           uci     -180    2.1     https://github.com/Warpten/Fruit-2.1
phalanx         xboard  -180    XXV     https://github.com/student-t/Phalanx-XXIII
fridolin3       uci     -200    3.1     https://sites.google.com/site/fridolinchess
drosophila      uci     -210    1.6     https://github.com/gustafullberg/drosophila             (cmake)
marvin          xb+uci  -220    5.0.0   https://github.com/bmdanielsson/marvin-chess
zevra2          uci     -370    2.1.2   https://github.com/sovaz1997/Zevra2             (/Zevra 1.8.6: -550)
taltos          xb+uci  -440    .       https://github.com/GBuella/Taltos
ct800           uci     -480    0.41    https://www.ct800.net/download.htm                      (make..sh)
greko-junior    xb+uci  -490    12.9    https://github.com/coderxlsn/greko-junior-chess
googleplex      uci     -500    1.7     https://gitlab.com/tsoj/Googleplex_Starthinker          (cmake)
sungorus        uci     -510    1.4     https://sites.google.com/site/sungorus
robocide        uci     -550    .       https://github.com/DanielWhite94/robocide
teki            uci     -600    .       https://github.com/Mk-Chan/Teki                         (cmake)
magic           xb+uci  -670    2       https://github.com/zzhangb4/MagiC-engine-chess
cinnamon        uci     -690    2.3.3   https://github.com/gekomad/Cinnamon
floyd           uci     -750    0.9     https://github.com/kervinck/floyd
chesley         xboard  -800    1.2.x   https://github.com/matthewgingell/chesley
snowy           uci     -850    0.2.5   https://github.com/JasonCreighton/snowy
gully2          xboard  -860    2.16    https://github.com/borriss/gully
purplehaze      xboard  -920    2.1.0   https://github.com/vinc/purplehaze
prophet4        xboard  -950    4       https://github.com/jswaff/prophet4
claudia         uci     -1000   0.5.1   https://github.com/antoniogarro/Claudia
sayuri          uci     -1000   18.5.23 https://github.com/MetalPhaeton/sayuri                  (cmake)
sbchess         uci     -1100   .       https://github.com/DoubleEspresso/chess
minnow          uci     -1250   150301  https://github.com/tm512/minnow

Monolith is broken. At least in Chess960. Both c-chess-cli and cutechess-cli agree. Compiled it normally from source, using 'make' (default target 'all').

Code: Select all

$ uname -srm
Linux 5.9.6-arch1-1 x86_64
$ cutechess-cli -each proto=uci tc=2+0.02 option.Hash=2 -engine cmd=../Engines/monolith_2.01 -engine cmd=../Engines/demolito -games 1920 -openings file=./test/chess960.epd -repeat -pgnout monolith.pgn -concurrency 8
[...]
Started game 253 of 1920 (Monolith 2 vs Demolito dev)
Warning: Illegal ponder move from Monolith 2: e4f4
Warning: Illegal ponder move from Monolith 2: f5f4
Warning: Illegal ponder move from Monolith 2: g8f8
[...]
Terminating process of engine Monolith 2(15)
Finished game 249 (Monolith 2 vs Demolito dev): 0-1 {White's connection stalls}
[...]
$ ./c-chess-cli -each tc=2+0.02 option.Hash=2 -engine cmd=../Engines/monolith_2.01 -engine cmd=../Engines/demolito -games 1920 -openings ./test/chess960.epd -repeat -pgn monolith.pgn -concurrency 8
[...]
[5] Started game 24 of 1920 (Demolito dev vs Monolith 2)
[7] engine Monolith 2 is unresponsive

c-chess-cli is able to crash it faster than cutechess-cli. That's a feature! You get smaller core dumbs to investigate

I suspect the reason is that c-chess-cli applies rule50 pruning:

• cutechess-cli sends 'position fen P0 moves M0', where P0 is the starting position from the EPD suite, and M0 are all the moves since the beginning of the game.
• c-chess-cli sends 'position fen P1 moves M1', where P1 is the last position in which rule50 was reset, and M1 are the moves since that position. This results in much shorter move lists on average, which is more practical for debugging engines (and possibly faster, though only mesurable in unrealisticly fast testing conditions).

For non-bogus engines, this makes no difference. But it's a great revealer of bogus engines:

• some engines don't don't parse 'moves' after 'fen' (but only after 'startpos').
• some engines make some wrong assumptions about the FEN in Chess960, that result in crashes when given a Chess960 FEN which is not a starting position (ie. only work if the Chess960 FEN has all pieces on 1st rank, and all pawns on 2nd rank, or whatever the specific bug may be).

[OliverBr]

Right, thanks! I guess this is probably about setup start position again -- in 4500+ games from standard starting position that I have on log GiuChess never made illegal moves (but made illegal draw claims, as mentioned before...).

I think GiuChess has the same issue as "ChessPuter":
It cannot handle openings forced by the GUI/TourneyManager. This is a most important feature.
Just give the engine a "third" color: BLACK, WHITE, NOENGINE. While being on "NOENGINE", just consume the next moves and play them without thinking for both colors. After "go", the engines takes the side, which is to move. (in CECP 1, there are also the commands "white" and "black", but they are not necessary in UCI or CECP 2).

[jdart]

Many engines indeed are not stable or do not handle fast time controls.

I am not clear why Arasan did not make your list? I routinely run tens of thousands of test games with it, commonly on Linux.

[unserializable]

Manchester is a most interesting engine and I hadn't tested it before: It compiles easily on both systems and is very stable.

Thanks, Oliver! Tests that weed out so many engines must be good, so this is reassuring that I managed to get many things in the protocol implementation correct. One thing I realized when passing the reference to build instructions is that compilation could be made bit simpler still, just by having 'release' build target be default in the Makefile (instead of the debug build target).

But there is still a issue:
It cannot handle fast time controls. It loses many times on a 40/25 time control (25 seconds for 40 moves). It would make it into the second list (candidates with one minor flaw).
...

Thanks again, this description is detailed and enough to address this flaw. At fixed depth 4, this version was also meant as solid reference point, engine which play remains constant strength (as long as it has time to think, obviously), so only way it breaks out of search is by ending it and search depth is only decreased when 'sd' sets it lower than 4. So time-losses on ultrabullet controls on current hardware are indeed an issue. I am not yet quite certain whether I *do* want to do something about that in version 1.0 point release or would be content to leave this flaw in 1.0 and leave it for 2.0. Will let you know when I do tackle the ultrabullet handling for Monchester .

[abulmo2]

2) Easy compile with make/cmake on both, Linux and MacOSX, with g++ and clang, telling their name on init.

You should have make it clear in your title that you restricted your list to C/C++ engines. People looking at your list may think programs in Rust, D, C#, java, etc. are automatically buggy.

[hgm]

Fairy-Max did not qualify? Or N.E.G. 1.2?

Fairy-Max was actually a hot candidate and I was very surprised to see it having crashed. It was version 4.8 with the source from here: https://home.hccnet.nl/h.g.muller/dwnldpage.html

I see now that there is Version 5 still to check out. If you tell me the the link to a reliable source, I will do it. Same for N.E.G.

Fairy-Max 4.8 never really crashes, but it can fall into excessively long thinking (like 30 min in a bullet game) at the moment it switches off null move (in the root), and then tries to improve the search depth it finds in the hash entry for the root by 1 ply without null move. This of course makes it forfeit on time. I never bothered to fix that, because it almost exclusively happens in lost positions, so that no Elo would be gained by the fix.

This should be cured in version 5, which invalidates its hash table before any search where the use of null move was switched.

The latest source of Fairy-Max can be found in my git repository:

http://hgm.nubati.net/cgi-bin/gitweb.cgi

Just download the latest snapshot of the main branch. Same for KingSlayer / simple.

[abulmo2]

I wonder what is wrong with Crafty to not be in your list.

[MikeB]

Did you consider Shallow Blue 2.0 ? I have run thousands of games without issue.