Cutechess-cli: A command line tool for engine-engine matches

Discussion of anything and everything relating to chess playing software and machines.

Moderator: Ras

User avatar
ilari
Posts: 750
Joined: Mon Mar 27, 2006 7:45 pm
Location: Finland

Re: Cutechess-cli: A command line tool for engine-engine mat

Post by ilari »

Graham Banks wrote:Hi Ilari,

amongst all your hard work on this, how is Sloppy coming along? :P

Cheers, Graham.
Hi Graham, and thanks for your interest.

One of the main reasons for writing Cutechess-cli was to have a powerful tool for testing Sloppy. The private version of Sloppy already has a parallel search, which I'm still testing. The evaluation function will be automatically tuned with a private version of Cutechess-cli that runs fast games and changes the weights with a genetic algorithm after each cycle.

I've also thought about doing what Tord did with Glaurung, and rewriting Sloppy in C++. Then I could more easily abstract the annoying low-level parts away.

It's going to take a while before I get to release anything. I'll have a bit more spare time during the summer, but I'm going to be training and playing in a lot of tennis tournaments, so I don't know if I'll have time for Sloppy.
User avatar
ilari
Posts: 750
Joined: Mon Mar 27, 2006 7:45 pm
Location: Finland

Re: Cutechess-cli: A command line tool for engine-engine mat

Post by ilari »

Matthias Gemuh wrote:
ilari wrote: - d7d8=q style promotions are now accepted, even though they violate the protocol. At least Thinker sends promotions in this format.

Hi Ilari,
I have caught you there :D :) :D :)
I will wait and see what comes next :wink:
It is sad that we must do such things.
regards,
Matthias.
Damn! I know, I kinda broke my promise there. But this was a very simple change, and I also did it because Thinker is a popular, strong, and stable engine. I'm still a bit annoyed that people who write brilliant engines can't get such a simple thing as the move format right.
User avatar
Matthias Gemuh
Posts: 3245
Joined: Thu Mar 09, 2006 9:10 am

Re: Cutechess-cli: A command line tool for engine-engine mat

Post by Matthias Gemuh »

ilari wrote:
Matthias Gemuh wrote:
ilari wrote: - d7d8=q style promotions are now accepted, even though they violate the protocol. At least Thinker sends promotions in this format.

Hi Ilari,
I have caught you there :D :) :D :)
I will wait and see what comes next :wink:
It is sad that we must do such things.
regards,
Matthias.
Damn! I know, I kinda broke my promise there. But this was a very simple change, and I also did it because Thinker is a popular, strong, and stable engine. I'm still a bit annoyed that people who write brilliant engines can't get such a simple thing as the move format right.
There are many strong version of Scorpio that people want to continue using.
You will need to deal with their move format too.
When I slackened protocol handling in ChessGUI, I stopped keeping track of which strong engine is breaking protocol and how.
Some engines below elo 2000 seem not to even know that there is a protocol :evil:

regards,
Matthias.
My engine was quite strong till I added knowledge to it.
http://www.chess.hylogic.de
User avatar
ilari
Posts: 750
Joined: Mon Mar 27, 2006 7:45 pm
Location: Finland

Re: Cutechess-cli: A command line tool for engine-engine mat

Post by ilari »

I released Cutechess-cli version 0.1.5

- A new "initstr" engine option for sending initialization strings to engines. Run "cutechess-cli --help for details"

- Allow "min" and "max" to have the same value in UCI options. This doesn't really affect anything, just removes a warning.

- Handle forfeits, especially timeouts, better. Previously engines got sometimes jammed and stopped responding to input. Hopefully this fixes the "engine failed to respond to ping" errors and timer warnings that happens with some engines in fast games.
Dave Gomboc

Re: Cutechess-cli: A command line tool for engine-engine mat

Post by Dave Gomboc »

ilari wrote:Damn! I know, I kinda broke my promise there. But this was a very simple change, and I also did it because Thinker is a popular, strong, and stable engine. I'm still a bit annoyed that people who write brilliant engines can't get such a simple thing as the move format right.
Why not shim such engines: write a thin adapter than takes in the output of non-standards-compliant engines and emits standards-compliant output? That adapter can load the broken engine and communicate with it: your application only interacts with the shim (and the other contestant).

Dave
User avatar
ilari
Posts: 750
Joined: Mon Mar 27, 2006 7:45 pm
Location: Finland

Re: Cutechess-cli: A command line tool for engine-engine mat

Post by ilari »

Dave Gomboc wrote: Why not shim such engines: write a thin adapter than takes in the output of non-standards-compliant engines and emits standards-compliant output? That adapter can load the broken engine and communicate with it: your application only interacts with the shim (and the other contestant).

Dave
We've thought about this.

More specifically, what we thought about was using scripting (QtScript in Cute Chess' case) for parsing the engine output. So when installing an incompatible engine the user could write a small script to convert the engine's output to proper Xboard engine's output.

We're still at the stage where we're trying to make sure that at least fully compatible Xboard and Winboard engines work flawlessly. For now, incompatible engines are not a high priority.
User avatar
ilari
Posts: 750
Joined: Mon Mar 27, 2006 7:45 pm
Location: Finland

Re: Cutechess-cli: A command line tool for engine-engine mat

Post by ilari »

Version 0.1.6 is out. This is mainly a bugfix release:

- Complete games are not played from opening books. It's still a good idea to limit book depth with the -bookdepth option if you use a really deep book (or PGN file as a book).

- If an engine becomes unresponsive (doesn't reply to the "ping" or "isready" command), it's terminated.

- If an engine doesn't quit after the "quit" command, it's terminated forcefully.

- Fix a serious race condition on Windows. If you've gotten "engine failed to respond to ping" warnings, timeouts, etc. it was likely because of this bug. Big thanks to Werner Schüle for helping me find the bug.
Edsel Apostol
Posts: 803
Joined: Mon Jul 17, 2006 5:53 am
Full name: Edsel Apostol

Re: Cutechess-cli: A command line tool for engine-engine mat

Post by Edsel Apostol »

ilari wrote:Version 0.1.6 is out. This is mainly a bugfix release:

- Complete games are not played from opening books. It's still a good idea to limit book depth with the -bookdepth option if you use a really deep book (or PGN file as a book).

- If an engine becomes unresponsive (doesn't reply to the "ping" or "isready" command), it's terminated.

- If an engine doesn't quit after the "quit" command, it's terminated forcefully.

- Fix a serious race condition on Windows. If you've gotten "engine failed to respond to ping" warnings, timeouts, etc. it was likely because of this bug. Big thanks to Werner Schüle for helping me find the bug.
Thanks Ilari. This version works well now without the "ping" problems. Its a big help to my testing. Looking forward to later versions with tournament support for easier setup of engine matches.
User avatar
ilari
Posts: 750
Joined: Mon Mar 27, 2006 7:45 pm
Location: Finland

Re: Cutechess-cli: A command line tool for engine-engine mat

Post by ilari »

Edsel Apostol wrote: Thanks Ilari. This version works well now without the "ping" problems. Its a big help to my testing. Looking forward to later versions with tournament support for easier setup of engine matches.
There are still a couple of bugs related to ping responses, specifically ending the game. Deuterium is one engine that currently gives me problems, but I'll fix that for the upcoming version.

Once Cutechess-cli is stable enough, I'll focus more on the GUI side: implementing human player, widgets for setting the variant, time control, better engine configuration, etc.
User avatar
ilari
Posts: 750
Joined: Mon Mar 27, 2006 7:45 pm
Location: Finland

Re: Cutechess-cli: A command line tool for engine-engine mat

Post by ilari »

Just uploaded version 0.1.7:

- Fixed an omission in the help text
- Tell engines to stop thinking when the game ends (if they are thinking)
- Put Xboard engines in force mode before sending them a move that ends the game. This is to make sure that the engine doesn't respond to a move when the game is over.
- New option (-wait) for setting a time delay between games. This is rarely needed because Cute Chess is pretty good at deciding when it's safe to start a new game.