Cutechess-cli: A command line tool for engine-engine matches

[ilari]

I've previously told that I was writing a cross-platform, cross-chess protocol GUI with a friend, now called CuteChess. The project is going strong: the GUI part is still early in development, but the core - the chess library for running games, handling PGN files, opening books, etc., is quite mature.

So as a side project I decided to write Cutechess-cli, which uses the same chess library we develop, but is only used for running chess matches from the command line, very much like Xboard. So why would anyone use this tool over Xboard?:

- No GUI means that the host doesn't need an Xserver (Unix). This is a very good thing if you use a cluster of headless PCs for engine testing.
- Lightweight. Uses very little memory and very few cpu cycles.
- Supports both Xboard and UCI


If you run cutechess-cli with the --help parameter you'll get this:

Code: Select all

Usage: cutechess-cli -fcp [eng_options] -scp [eng_options] [options]
Options:
  -fcp <options>	Apply <options> to the first engine
  -scp <options>	Apply <options> to the second engine
  -both <options>	Apply <options> to both engines
  -variant <arg>	Set chess variant to <arg>. Must be Standard,
			Fischerandom, Capablanca, Gothic or Caparandom
  -book <file>		Use <file> &#40;Polyglot book file&#41; as the opening book
  -bookdepth <n>	Set the maximum book depth &#40;in plies&#41; to <n>
  -event <arg>		Set the event name to <arg>
  -games <n>		Play <n> games
  -debug		Display all engine input and output
  -pgnin <file>		Use <file> as the opening book in PGN format
  -pgnout <file>	Save the games to <file> in PGN format
  -repeat		Play each opening twice so that both players get
			to play it on both sides
  -site <arg>		Set the site/location to <arg>

Engine options&#58;
  name=<arg>		Set the name to <arg>
  cmd=<arg>		Set the command to <arg>
  dir=<arg>		Set the working directory to <arg>
  proto=<arg>		Set the chess protocol to <arg>. Must be xboard or uci
  tc=<arg>		Set the time control to <arg>. The format is
			moves/time+increment, where 'moves' is the number of
			moves per tc, 'time' is time per tc &#40;either seconds or
			minutes&#58;seconds&#41;, and 'increment' is time increment
			per move in seconds
  cpus=<n>		Tell the engine to use a maximum of <n> cpus
  egbbpath=<dir>	Set the path to endgame bitbases to <dir>
  egtbpath=<dir>	Set the path to endgame tablebases to <dir>
  uci/<name> <arg>	Set UCI option <name> to value <arg>

That's probably not sufficient documentation, so I'll give a couple of examples:

cutechess-cli -fcp name=Atak cmd=Atak32.exe dir=C:\chess\atak proto=xboard -scp cmd=glaurung proto=uci cpus=1 -both tc=40/60 -games 2

- Use the "name=Atak" parameter because it's a Xboard protocol 1 engine and doesn't tell its name.
- Use the "dir=C:\chess\atak" parameter because Atak's executable isn't in the PATH environment.
- Glaurung can tell its name and is in the path, so I don't use the name and dir parameters.
- Set Glaurung to use 1 thread
- Set the time control for both players to 40 moves in 60 seconds
- Play 2 games


cutechess-cli -fcp cmd=sloppy proto=xboard tc=40/1:30+1 -fcp cmd=glaurung proto=uci tc=60 uci/Hash=64 -games 10 -debug -book C:\dann_books\book.bin -bookdepth 20 -repeat -pgnout games.pgn

- Set Sloppy's time control to 40 moves in 1 minutes and 30 seconds with a 1 second increment.
- Set Glaurung's time control to 60 seconds for the whole game.
- Set Glaurung's Hash option to 64. You can put any option name after "uci/".
- Play 10 games.
- The -debug option means that all engine input and output is displayed. If you want to log it in a file you can put "> logfile.txt" to the end of the command.
- Use "C:\dann_books\book.bin" as the opening book. It must be in Polyglot format.
- Play a maximum of 20 halfmoves from the book.
- Play each opening (book line) twice, so that both engines get to play it on both sides
- Save the games to "games.pgn"


Those who like to compile stuff can grab the source code here: http://repo.or.cz/w/sloppygui.git

I also have binaries for some platforms:

Mac OSX 10.4 and up (Intel only): http://koti.mbnet.fi/~ilaripih/bin/cute ... li-osx.zip
Use the "cutechess-cli.sh" script, not the binary file directly.

Linux 64-bit: http://koti.mbnet.fi/~ilaripih/bin/cute ... x64.tar.gz
You need libqt4-core installed ("sudo apt-get install libqt4-core" on Ubuntu)
Use the "cutechess-cli.sh" script, not the binary file directly.

Windows 32-bit: http://koti.mbnet.fi/~ilaripih/bin/cute ... -win32.zip

I haven't tested the binaries on machines that don't already have Qt libraries installed, so I hope they work. If not, please let me know.
I don't have a binary for 32-bit Linux yet.


This is work in progress, so there may be bugs. I'd appreciate it if you could report them to me.

[Laurens Winkelhagen]

Hi,

This sounds like a nice tool:-) I'm wondering, is there any planned support for chessservers like FICS? Your tool could also be a great for managing a computer account there on some old laptop w/o screen and little memory for xwindows (which I happen to have^^)

Thanks!

[ilari]

I'm wondering, is there any planned support for chessservers like FICS?

FICS support is on my todo list. From now on the GUI side is going to be my main focus, but as we develop the core library, Cutechess-cli will benefit from it as well.

Cutechess-cli is actually a very small project, just a few hundred lines of C++ code. The power behind it is in the chess library which anyone can easily use as a base for their own GUI, engine-testing tool, opening book editor, PGN editor, etc.

[Zach Wegner]

Hello Ilari,

Thanks a bunch for providing this tool, and again for making it open source. It looks to be a very good solution for me for background testing, and will make a good complement to xboard (for live viewing).

[ilari]

I forgot to mention some of the features that Cutechess-cli currently has and doesn't have:

- Supports standard chess, Chess960, Capablanca, Gothic, and Capablanca random variants
- Supports Polyglot opening books
- Can use PGN collections as opening books
- White's time control can be different from black's

- No pondering
- No analysis support
- The "edit" command from Xboard protocol 1 isn't supported. So it's not possible to send custom FEN strings to XB1 engines.


It's possible to start the games from custom FEN strings by putting them in a PGN file and using the "-pgnin <file>" option. The PGN file would then look like this:

Code: Select all

[FEN "fen 1"]
*

[FEN "fen 2"]
*

[FEN "fen n"]
*

The termination marker (*) is needed. Without it the PGN parser wouldn't know where one game ends and another one begins.

Btw, has anyone gotten this thing to work yet? I found the deployment and dependency hell really annoying, so I wouldn't be surprised if something was missing from my binary packages.

[ilari]

Hello Ilari,

Thanks a bunch for providing this tool, and again for making it open source. It looks to be a very good solution for me for background testing, and will make a good complement to xboard (for live viewing).

Thanks. I needed this tool myself for testing the core functionality of our GUI, and for testing Sloppy. I've found it to work fine even for lightning-fast games (eg. 1 second per game), which Xboard/Winboard can't handle that well.

[Marc Lacrosse]

Thanks a lot Ilari : this looks very nice and I hope I will soon have time for testing it on windows.

A little suggestion : maybe it could be interesting to add adjudication/resign features.

Something like :

-DrawAdj x y
where x = move number, y = centipawns evaluation
Game is adjudicated as a draw at move number "x" if absolute value of both engines evaluation is less than "y" centipawns.

- Resign v w
where v = number of consecutive moves, w = centipawn evaluation
An engine is forced to resign if its evaluation is less than "w" for at least "v" consecutive moves.

Marc

[ilari]

Thanks a lot Ilari : this looks very nice and I hope I will soon have time for testing it on windows.

A little suggestion : maybe it could be interesting to add adjudication/resign features...

Good suggestions, and quite easy to implement. I'll also need to add a -invertscore option for Xboard engines that always print the score from white's perspective, like Crafty.

[IanO]

Doesn't work for UCI engines on OS X 10.4

$ ./cutechess-cli.sh -fcp cmd=crafty proto=xboard -scp cmd=homer "dir=/Applications/Games/Sigma Chess 6.1/Homer" proto=uci -both tc=60/300 -pgnout crafty-homer.pgn -debug
Warning: Cannot start engine "homer"
Warning: QProcess: Destroyed while process is still running.

I confirmed that I can start UCI engines from within their home directory.

[Zach Wegner]

Doesn't work for UCI engines on OS X 10.4

$ ./cutechess-cli.sh -fcp cmd=crafty proto=xboard -scp cmd=homer "dir=/Applications/Games/Sigma Chess 6.1/Homer" proto=uci -both tc=60/300 -pgnout crafty-homer.pgn -debug
Warning: Cannot start engine "homer"
Warning: QProcess: Destroyed while process is still running.

I confirmed that I can start UCI engines from within their home directory.

Change "cmd=homer" to "cmd=./homer", and make sure you have execute permissions on the engine. Just figured this out myself about 15 minutes ago.