Cutechess-cli: A command line tool for engine-engine matches

[ilari]

Another problem: in a match between Hamsters and Crafty 20.14, several of the games were lost by crafty due to an illegal first move, which turns out to be a legal move in the preceding drawn game (once by 50-move rule, twice by threefold repetition). Looks like a race condition between starting a new game and waiting for the previous program to quit the previous game.

Thanks for the report. It would be easy to tackle this problem by synchronizing the engine with cutechess-cli at the end of a game with the ping command, but since all engines don't support it, I'm going to have to come up with something better.

Stay tuned for the next version.

[ilari]

As promised, I've released a new version (0.1.1) of Cutechess-cli. Links to the binaries are in the first post. All issues that were reported to me should be fixed, and most of the features requested are there as well:

- On Linux/OSX the startup script is fixed so one can use paths with spaces in the "dir" engine option, as long as they're wrapped in quotes of course.

- It's possible to include the full path to the executable in the "cmd" engine option, even if it contains spaces.

- There's a new "arg" engine option which allows the user to pass arguments to engines. This option can be used multiple times. So to set Crafty's book path one would type: 'cmd=crafty arg="bookpath=/opt/local/share/crafty"'. To run a Windows engine under wine in Linux, use 'cmd=wine arg=engine.exe'.

- Cutechess-cli now makes sure that engines have finished playing before a new game is started. Every engine that supports the ping command shoud work without problems. Of the rest, at least 90% should work, but there could be special cases.

- There are now engine options to set limits to search depth and node count (run cutechess-cli --help to learn more about them). Currently the node limit works only for UCI engines. A time control must still be specified when these limits are used. So if you don't want the time control to affect the search time, just set it to a very big value.

- As suggested by Marc Lacrosse, I added draw and resign adjudication options. A couple of examples:
-- Adjudicate as draw if both engines give a score between -10 and +10 centipawns after 250 fullmoves: "-draw 250 10".
-- Adjudicate as loss when an engine's score is at least 500 centipawns below zero for at least 5 consecutive moves: "-resign 5 500".

I implemented the -resign option this way because passing a negative value would have the "-" sign, and would be considered a separate command line argument.

- There's a "invertscores" engine option that should be used with engines that report evaluation scores from white's point of view. Crafty is the only one I know of. If you use the resign adjudication it's very important that the "invertscores" option is used when applicable.


I tested all the new features and fixes with several engines (Xb1, Xb2, UCI) and on all three supported platforms, but it may not have been extensive enough. So again, I'd appreciate any feedback. I always take bug reports and feature requests seriously.

[Edsel Apostol]

Thanks Ilari for including the node limit for UCI engines.

How about adding tournament support? For example there will be a text file that has the tournament settings. Then the user can play a tournament and if he chooses to stop the tournament, the current results should be saved and the tournament can be resumed on another time. The tournament settings should easily be editable like the opponents, time control, starting positions, etc.

[ilari]

Thanks Ilari for including the node limit for UCI engines.

No problem, that was an easy one.

How about adding tournament support? For example there will be a text file that has the tournament settings. Then the user can play a tournament and if he chooses to stop the tournament, the current results should be saved and the tournament can be resumed on another time. The tournament settings should easily be editable like the opponents, time control, starting positions, etc.

We are going to support tournaments, but that's going to take more time to do. I've already designed an XML file format for tournaments. The next step is to write the Tournament class, the configuration model and other supporting classes, and put them in the chess library. Then we can use the same tournament code in GUI and CLI side. So it will be possible to set up a tournament in the GUI and then run it with Cutechess-cli.

[Tord Romstad]

Hello Ilari,

This looks like a useful tool -- thanks for sharing!

I just gave the Mac OS X version a quick try, and tried running a 1000 game match between two experimental Glaurung versions at 40 moves/1 second (which may seem like a crazy time control, but keep in mind that this program is written to run on a machine 100 times slower than my computer). After 687 games, the program crashed with the following error message:

Code: Select all

Started game 688 of 1000
./cutechess-cli.sh: line 13: 78392 Bus error               $dirname/$appname "$@"

Tord

[ilari]

Hello Ilari,

This looks like a useful tool -- thanks for sharing!

I just gave the Mac OS X version a quick try, and tried running a 1000 game match between two experimental Glaurung versions at 40 moves/1 second (which may seem like a crazy time control, but keep in mind that this program is written to run on a machine 100 times slower than my computer). After 687 games, the program crashed with the following error message:

Code: Select all

Started game 688 of 1000
./cutechess-cli.sh: line 13: 78392 Bus error               $dirname/$appname "$@"

Tord

Thanks for report. I managed to reproduce the problem with my debug build which aborted on an assertion. The crash happened when one of the engines lost on time.

The problem should be fixed now. I've updated the binaries and commited the fix to our source code repository.

[bob]

Hello Ilari,

Thanks a bunch for providing this tool, and again for making it open source. It looks to be a very good solution for me for background testing, and will make a good complement to xboard (for live viewing).

I suppose I should release my code at some point, but it is really tied to unix. It is what I use on our cluster and will handle using multiple starting positions, play as many games per position as you want, alternating colors (I use 2 here), and creates a PGN file containing all the games with results. I have another program that will take a bunch of scripts that use the above program, and execute as many at once as you want, so that you can use all your CPUs or just 1/2 of them or whatever. But again, all is pure Unix-based, no windows at all, and documentation is <nil>

[Edsel Apostol]

I've tried the fixed node match and it worked well. Great tool!

By the way, it would be good if you could support enabling/disabling the depth, time and score in the PGN in order to save storage space.

[Zach Wegner]

Hello Ilari,

Another little bug report. I actually noticed this because I was setting the time control to 1+0, which turns out to be 1 second/game. I think that's rather confusing, as it breaks with the xboard protocol (which would specify one second as 0:01). But anyways, it sends the level command to the engine without 0-padding the time, so the engine receives "level 0 0:1 0". I traced the cause to this function, hoping I could fix it, but I know nothing about Qt.

Code: Select all

static QString msToXboardTime(int ms)
{
    int sec = ms / 1000;

    QString number = QString::number(sec / 60);
    if (sec % 60 != 0)
        number += QString(":%1").arg(sec % 60);

    return number;
}

One other request: could you put an interrupt handler in the program? This is a pretty minor issue, but sometimes I need to kill the program, and the logfile just abruptly ends now. It would be nice to have a clean break there, and maybe save the first part of the game in the pgn file.

Thanks,
Zach

[Edsel Apostol]

I want to add that Glaurung 2.2 sometimes makes an illegal move around 3 to 5 times in 60 games. This is my configuration:

Code: Select all

cutechess-cli -fcp cmd=glaurung-w32.exe dir=D:\chess\glaurung2.2\Windows\ uci/Threads=1 -scp cmd=t20090325.exe dir=D:\chess\src20090325\ -both proto=uci tc=1/300+0 nodes=8192 uci/Hash=32 -draw 120 10 -resign 5 300 -games 60 -pgnin D:\chess\NoomenTestsuite2008.pgn -repeat -pgnout games.pgn

I haven't tried yet using -debug to look at the matter but here's what you can find in the PGN file:

Code: Select all

[Event "?"]
[Site "?"]
[Date "2009.03.25"]
[Round "9"]
[White "Glaurung 2.2 JA"]
[Black "Twisted Logic 20090324z"]
[Result "0-1"]
[PlyCount "30"]
[TimeControl "1/300"]

1. e4 {book} e5 {book} 2. Nf3 {book} Nc6 {book} 3. Bb5 {book} a6 {book}
4. Ba4 {book} Nf6 {book} 5. O-O {book} Be7 {book} 6. Re1 {book} b5 {book}
7. Bb3 {book} d6 {book} 8. c3 {book} O-O {book} 9. h3 {book} Nb8 {book}
10. d4 {book} Nbd7 {book} 11. Nbd2 {book} Bb7 {book} 12. Bc2 {book} Re8 {book}
13. Nf1 {book} Bf8 {book} 14. Ng3 {book} g6 {book} 15. a4 {book} c5 {book}
{White made an illegal move: (none)} 0-1

Note that the time control is just a hack to support fixed nodes. This was run using fixed nodes of 8192.