Cutechess-cli: A command line tool for engine-engine matches

Discussion of anything and everything relating to chess playing software and machines.

Moderators: hgm, Rebel, chrisw

mcostalba
Posts: 2684
Joined: Sat Jun 14, 2008 9:17 pm

Re: Cutechess-cli: A command line tool for engine-engine mat

Post by mcostalba »

ilari wrote:
Gian-Carlo Pascutto wrote:Next up the wish-list: SMP support. I know you can launch separate instances, but the tournaments don't necessarily stop at the same time.
I've already planned support for concurrent tournament games, once tournaments have been implemented. If you wish I can add concurrency support for matches between two engines as well.
Hi Ilari,

I would be very interested in concurrency support for matches between two engines.

We would like to start as example 8 threads in parallel so to play 4 games, each one between two versions of the same engine.

Currently we need to start 4 separate instances of cutechess, but this is not an optimal solution because you need to group the results at the end, while if cutechess would be able to handle that you have only one pgn file and one results output. And also you have to create 4 directories with engines, books, etc..

We cannot say is an elegant solution ;-)


I have cloned the latest repo but I didn't see this feature. Are you planning to support that ?

Thanks
Marco
Edsel Apostol
Posts: 803
Joined: Mon Jul 17, 2006 5:53 am
Full name: Edsel Apostol

Re: Cutechess-cli: A command line tool for engine-engine mat

Post by Edsel Apostol »

mcostalba wrote:
ilari wrote:
Gian-Carlo Pascutto wrote:Next up the wish-list: SMP support. I know you can launch separate instances, but the tournaments don't necessarily stop at the same time.
I've already planned support for concurrent tournament games, once tournaments have been implemented. If you wish I can add concurrency support for matches between two engines as well.
Hi Ilari,

I would be very interested in concurrency support for matches between two engines.

We would like to start as example 8 threads in parallel so to play 4 games, each one between two versions of the same engine.

Currently we need to start 4 separate instances of cutechess, but this is not an optimal solution because you need to group the results at the end, while if cutechess would be able to handle that you have only one pgn file and one results output. And also you have to create 4 directories with engines, books, etc..

We cannot say is an elegant solution ;-)


I have cloned the latest repo but I didn't see this feature. Are you planning to support that ?

Thanks
Marco
I am also interested in this feature. In my testing I only used one pgn output for 4 matches running at the same time and in around 1200 games there would be around 1 or 2 games missing as they might be written at the same time by one of the cutechess instances. If concurrency is supported I think that it wouldn't be an issue anymore.

It would also be good to have support for epd as starting positions. I can't seem to find this feature if its already been implemented.

And another feature request is the ability to turn off the computer after a match or a tournament just like in Arena.
User avatar
ilari
Posts: 750
Joined: Mon Mar 27, 2006 7:45 pm
Location: Finland

Re: Cutechess-cli: A command line tool for engine-engine mat

Post by ilari »

mcostalba wrote:
ilari wrote:
Gian-Carlo Pascutto wrote:Next up the wish-list: SMP support. I know you can launch separate instances, but the tournaments don't necessarily stop at the same time.
I've already planned support for concurrent tournament games, once tournaments have been implemented. If you wish I can add concurrency support for matches between two engines as well.
Hi Ilari,

I would be very interested in concurrency support for matches between two engines.

We would like to start as example 8 threads in parallel so to play 4 games, each one between two versions of the same engine.
Hi,
Something like that is coming. We've both been busy this summer, so development has been somewhat slow. But we have designed a GameManager class that can handle multiple concurrent and/or queued games, and it will be used for both engine-engine matches and tournaments.

If we were just developing an application, Cute Chess 1.0 would probably be finished already. But we're also trying to write a powerful, yet easy to use library that others can use to make their own chess tools, guis, etc. That's taking most of the time, because we have to make it easy to extend the library, provide a very high abstraction level, complete API documentation, maybe even binary compatibility between different versions, etc.
User avatar
ilari
Posts: 750
Joined: Mon Mar 27, 2006 7:45 pm
Location: Finland

Re: Cutechess-cli: A command line tool for engine-engine mat

Post by ilari »

Edsel Apostol wrote:It would also be good to have support for epd as starting positions. I can't seem to find this feature if its already been implemented.
That's on my TODO list, but there are many high-priority tasks that we'll have to do first.

And another feature request is the ability to turn off the computer after a match or a tournament just like in Arena.
That's easy, but also annoyingly platform-specific, so we'll see what we can do.
Tord Romstad
Posts: 1808
Joined: Wed Mar 08, 2006 9:19 pm
Location: Oslo, Norway

Re: Cutechess-cli: A command line tool for engine-engine mat

Post by Tord Romstad »

Edsel Apostol wrote:It would also be good to have support for epd as starting positions. I can't seem to find this feature if its already been implemented.
Just write a little program which takes an EPD file as input and spits out a PGN file. I use the following Haskell two-liner:

Code: Select all

pgnify = (\x -> "[FEN "" ++ x ++ ""]\n*\n") . unwords . take 4 . words

main = interact $ unlines . map pgnify . lines
Use like this:

Code: Select all

cat silver.epd | epdtopgn > silver.pgn
mcostalba
Posts: 2684
Joined: Sat Jun 14, 2008 9:17 pm

Re: Cutechess-cli: A command line tool for engine-engine mat

Post by mcostalba »

ilari wrote: Hi,
Something like that is coming. We've both been busy this summer, so development has been somewhat slow. But we have designed a GameManager class that can handle multiple concurrent and/or queued games, and it will be used for both engine-engine matches and tournaments.

If we were just developing an application, Cute Chess 1.0 would probably be finished already. But we're also trying to write a powerful, yet easy to use library that others can use to make their own chess tools, guis, etc. That's taking most of the time, because we have to make it easy to extend the library, provide a very high abstraction level, complete API documentation, maybe even binary compatibility between different versions, etc.
Thanks !

I have seen from the git log that you have rewritten a lot of stuff lately ;-)
User avatar
ilari
Posts: 750
Joined: Mon Mar 27, 2006 7:45 pm
Location: Finland

Re: Cutechess-cli: A command line tool for engine-engine mat

Post by ilari »

mcostalba wrote:Thanks !

I have seen from the git log that you have rewritten a lot of stuff lately ;-)
Yes, and there's always more rewriting to do. For example the rewritten engine configuration classes will probably be rewritten again. But as long as the next iteration is better than the previous one, it's all good.
User avatar
ilari
Posts: 750
Joined: Mon Mar 27, 2006 7:45 pm
Location: Finland

Cutechess-cli version 0.1.8 released

Post by ilari »

It's been a loooong time since the last binary update and version number bump. However, a lot of changes were made between 0.1.7 and 0.1.8, most of them under the hood. These are the visible/interesting ones:

* Draw claims that happen on the opponent's turn are evaluated immediately
* When using PGN games as an opening book, the starting side is now set correctly according to the "FEN" tag (if any)
* Fixed a bug that caused some legal en-passant moves in SAN format to be rejected
* Fixed a bug that sometimes caused a crash when using SAN move format with an Xboard engine (like Crafty)
* Fixed a bug where cutechess-cli would try to terminate an already terminated engine
* Some optimizations, especially in move generation and parsing moves
* SIGINT (Ctrl-C) now terminates the engines forcefully
* Scale move time's decimal precision in PGN comments based on the time
* Support for Polyglot castling moves where the king "captures" a friendly rook
* Detect mate-in-n scores and mark them properly in PGN output. This works with all UCI engines and most Xboard engines.
* Parse Xboard engines' thinking output more flexibly
* Add a timestamp to every engine debug message
* The engine option code has been rewritten. As a result, generic engine options 'cpus', 'egtpath' and 'egbbpath' are no longer available (for now). "option.option_name=option_value" still works for both UCI and Xboard engines.
* The match won't be started if the opening book or PGN input file can't be read
* There's a README file in the package that explains all the options
* THE ABILITY TO RUN CONCURRENT GAMES!

The last feature is probably the most exciting one, and the most requested one. There's a new option, '-concurrency <n>' where <n> is the maximum number of games to run concurrently. The default is of course 1. Concurrency shouldn't cause any side effects in cutechess-cli - games are started in the right order and they're saved in PGN output in the right order. That said, I can't guarantee that the engines also won't have side effects - if there are multiple instances of an engine using the same learning, log, or other output file at the same time, ugly things can happen.

I've tested the concurrency feature with tens of thousands of superfast games on 64-bit Linux, and it seems to work. On Windows and Mac I didn't test much, so let's hope everything is fine on those platforms as well. I look forward to any feedback and bug reports.

As always, download links are in the first message of this thread. Some of the other information in that message may not be true/relevant anymore so I recommend reading the README file that comes with the program.
Dann Corbit
Posts: 12537
Joined: Wed Mar 08, 2006 8:57 pm
Location: Redmond, WA USA

Re: Cutechess-cli: A command line tool for engine-engine mat

Post by Dann Corbit »

It would be very nice if it could support USI also (there is no tournament GUI for USI at all that I know of, though you can play a single engine against a single engine for N games with Shogidokoro):
http://www.glaurungchess.com/shogi/usi.html
mcostalba
Posts: 2684
Joined: Sat Jun 14, 2008 9:17 pm

Re: Cutechess-cli version 0.1.8 released

Post by mcostalba »

Great job !!!!

Thanks a lot, especially for concurrency that is ver very welcomed :-)