Cutechess-cli 0.3.0 released

Discussion of anything and everything relating to chess playing software and machines.

Moderators: hgm, Harvey Williamson, bob

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
User avatar
ilari
Posts: 750
Joined: Mon Mar 27, 2006 5:45 pm
Location: Finland
Contact:

Cutechess-cli 0.3.0 released

Post by ilari » Thu Oct 07, 2010 5:52 pm

After half a year it's time to push out a new version of Cutechess-cli.

Changes include:
* The required Qt version is now 4.6 (was 4.5).
* Redesigned and rewritten concurrency code. The rare deadlocks and segfaults between games (especially with high "-concurrency" values) should be gone now.
* Fix the command line parser in the Windows version. Now command line arguments following the executable name are parsed properly.
* Handle invalid draw claims from Xboard engines properly. Now an invalid draw claim from an engine in "Thinking" mode is recognized properly and does not result in a "Connection stalls" error.
* Check the legality of opening book moves. It's no longer possible to break the game with a faulty book file.
* Option "-srand" for setting the random seed for the book move selector manually.
* The "invertscores" option was renamed to "whitepov".
* Evaluation perspective ("whitepov") and init strings can now be configured with the engines.json config file.
* Option "-recover" for automatically recovering crashed or stalled engines instead of terminating the whole match.
* Display the ELO difference after the match has ended.

We also have a mailing list now. I don't know if it works yet (Arto is administering it), but I hope it does. It should be a good place to announce new releases, report bugs and ask questions.


About the engines.json file:

Instead of using the "cmd", "name", "proto", "dir", "whitepov" and "initstr" arguments it's possible to keep engine configurations in a file and invoke an engine with just "conf=name". The file should be named "engines.json" and stored in the same folder with cutechess-cli. The chosen notation is JSON (JavaScript Object Notation), and the contents should look like this:

Code: Select all

[
   {
      "command":"crafty",
      "name":"Crafty-22.10",
      "protocol":"xboard",
      "whitepov":true
   },
   {
      "command":"wine \"Rybka v2.2n2.mp.w32.exe\"",
      "name":"Rybka-2.2n2",
      "protocol":"uci",
      "workingDirectory":"/home/ilari/chess/Rybka22n2"
   },
   {
      "command":"./sloppy",
      "initStrings" : [
         "init str 1",
         "init str 2"
      ],
      "name":"Sloppy-0.2.2",
      "protocol":"xboard",
      "workingDirectory":"/home/ilari/chess/sloppy",
   },
]
Remember to escape special characters with a backslash. So a double quote is \", a backslash is \\, etc.


Source code (cutechess-lib + cutechess-gui + cutechess-cli) here

Binaries:
* Linux 32-bit
* Linux 64-bit
* Mac OSX (Leopard or newer)
* Windows 32-bit

Have fun!

Ferdy
Posts: 4072
Joined: Sun Aug 10, 2008 1:15 pm
Location: Philippines

Re: Cutechess-cli 0.3.0 released

Post by Ferdy » Fri Oct 08, 2010 5:27 am

ilari wrote:After half a year it's time to push out a new version of Cutechess-cli.

Changes include:
* The required Qt version is now 4.6 (was 4.5).
* Redesigned and rewritten concurrency code. The rare deadlocks and segfaults between games (especially with high "-concurrency" values) should be gone now.
* Fix the command line parser in the Windows version. Now command line arguments following the executable name are parsed properly.
* Handle invalid draw claims from Xboard engines properly. Now an invalid draw claim from an engine in "Thinking" mode is recognized properly and does not result in a "Connection stalls" error.
* Check the legality of opening book moves. It's no longer possible to break the game with a faulty book file.
* Option "-srand" for setting the random seed for the book move selector manually.
* The "invertscores" option was renamed to "whitepov".
* Evaluation perspective ("whitepov") and init strings can now be configured with the engines.json config file.
* Option "-recover" for automatically recovering crashed or stalled engines instead of terminating the whole match.
* Display the ELO difference after the match has ended.

We also have a mailing list now. I don't know if it works yet (Arto is administering it), but I hope it does. It should be a good place to announce new releases, report bugs and ask questions.


About the engines.json file:

Instead of using the "cmd", "name", "proto", "dir", "whitepov" and "initstr" arguments it's possible to keep engine configurations in a file and invoke an engine with just "conf=name". The file should be named "engines.json" and stored in the same folder with cutechess-cli. The chosen notation is JSON (JavaScript Object Notation), and the contents should look like this:

Code: Select all

[
   {
      "command":"crafty",
      "name":"Crafty-22.10",
      "protocol":"xboard",
      "whitepov":true
   },
   {
      "command":"wine "Rybka v2.2n2.mp.w32.exe"",
      "name":"Rybka-2.2n2",
      "protocol":"uci",
      "workingDirectory":"/home/ilari/chess/Rybka22n2"
   },
   {
      "command":"./sloppy",
      "initStrings" : [
         "init str 1",
         "init str 2"
      ],
      "name":"Sloppy-0.2.2",
      "protocol":"xboard",
      "workingDirectory":"/home/ilari/chess/sloppy",
   },
]
Remember to escape special characters with a backslash. So a double quote is ", a backslash is \\, etc.


Source code (cutechess-lib + cutechess-gui + cutechess-cli) here

Binaries:
* Linux 32-bit
* Linux 64-bit
* Mac OSX (Leopard or newer)
* Windows 32-bit

Have fun!
* Option "-recover" for automatically recovering crashed or stalled engines instead of terminating the whole match.

Why not make this as default?
Would this mean that this will be applied only for suspecting engines?

Thank you again for this great tool. My development is very productive because of this.

Ferdy
Posts: 4072
Joined: Sun Aug 10, 2008 1:15 pm
Location: Philippines

Re: Cutechess-cli 0.3.0 released

Post by Ferdy » Fri Oct 08, 2010 5:47 am

ilari wrote:After half a year it's time to push out a new version of Cutechess-cli.

Changes include:
* The required Qt version is now 4.6 (was 4.5).
* Redesigned and rewritten concurrency code. The rare deadlocks and segfaults between games (especially with high "-concurrency" values) should be gone now.
* Fix the command line parser in the Windows version. Now command line arguments following the executable name are parsed properly.
* Handle invalid draw claims from Xboard engines properly. Now an invalid draw claim from an engine in "Thinking" mode is recognized properly and does not result in a "Connection stalls" error.
* Check the legality of opening book moves. It's no longer possible to break the game with a faulty book file.
* Option "-srand" for setting the random seed for the book move selector manually.
* The "invertscores" option was renamed to "whitepov".
* Evaluation perspective ("whitepov") and init strings can now be configured with the engines.json config file.
* Option "-recover" for automatically recovering crashed or stalled engines instead of terminating the whole match.
* Display the ELO difference after the match has ended.

We also have a mailing list now. I don't know if it works yet (Arto is administering it), but I hope it does. It should be a good place to announce new releases, report bugs and ask questions.


About the engines.json file:

Instead of using the "cmd", "name", "proto", "dir", "whitepov" and "initstr" arguments it's possible to keep engine configurations in a file and invoke an engine with just "conf=name". The file should be named "engines.json" and stored in the same folder with cutechess-cli. The chosen notation is JSON (JavaScript Object Notation), and the contents should look like this:

Code: Select all

[
   {
      "command":"crafty",
      "name":"Crafty-22.10",
      "protocol":"xboard",
      "whitepov":true
   },
   {
      "command":"wine "Rybka v2.2n2.mp.w32.exe"",
      "name":"Rybka-2.2n2",
      "protocol":"uci",
      "workingDirectory":"/home/ilari/chess/Rybka22n2"
   },
   {
      "command":"./sloppy",
      "initStrings" : [
         "init str 1",
         "init str 2"
      ],
      "name":"Sloppy-0.2.2",
      "protocol":"xboard",
      "workingDirectory":"/home/ilari/chess/sloppy",
   },
]
Remember to escape special characters with a backslash. So a double quote is ", a backslash is \\, etc.


Source code (cutechess-lib + cutechess-gui + cutechess-cli) here

Binaries:
* Linux 32-bit
* Linux 64-bit
* Mac OSX (Leopard or newer)
* Windows 32-bit

Have fun!
problem with the first sample run with this version.

Started game 1 of 100
Game 1 ended: 0-1 {Black wins by adjudication}
Hamsters 0.7.1 wins the game as black
Score of XboardEngine vs Hamsters 0.7.1: 0 - 1 - 0

First engine was not identified in the first game, other games are ok.
I did not use the option name, concurrency is set to 1.

This probem did not happen on previous verions 0.2.2.

User avatar
ilari
Posts: 750
Joined: Mon Mar 27, 2006 5:45 pm
Location: Finland
Contact:

Re: Cutechess-cli 0.3.0 released

Post by ilari » Fri Oct 08, 2010 1:56 pm

Ferdy wrote:* Option "-recover" for automatically recovering crashed or stalled engines instead of terminating the whole match.

Why not make this as default?
Would this mean that this will be applied only for suspecting engines?

Thank you again for this great tool. My development is very productive because of this.
The -recover option applies to any engine, even remote engines (eg. via SSH). It means that if an engine crashes, it loses the game and is restarted for the next game.

It's not a default because most people using cutechess-cli are testing unstable development versions of their engines, and they often want to halt the test run immediately when a crash happens. But if one just wants to find out which engine is stronger, A or B, then it's recommended to use the -recover option to make sure all games are played.

User avatar
ilari
Posts: 750
Joined: Mon Mar 27, 2006 5:45 pm
Location: Finland
Contact:

Re: Cutechess-cli 0.3.0 released

Post by ilari » Fri Oct 08, 2010 2:04 pm

Ferdy wrote:problem with the first sample run with this version.

Started game 1 of 100
Game 1 ended: 0-1 {Black wins by adjudication}
Hamsters 0.7.1 wins the game as black
Score of XboardEngine vs Hamsters 0.7.1: 0 - 1 - 0

First engine was not identified in the first game, other games are ok.
I did not use the option name, concurrency is set to 1.

This probem did not happen on previous verions 0.2.2.
Thanks for the report. I think I know why this happens now and didn't happen with v0.2.2. Is it possible that your engine sends feature done=1 before myname=engine_name?

Ferdy
Posts: 4072
Joined: Sun Aug 10, 2008 1:15 pm
Location: Philippines

Re: Cutechess-cli 0.3.0 released

Post by Ferdy » Fri Oct 08, 2010 4:54 pm

ilari wrote:
Ferdy wrote:problem with the first sample run with this version.

Started game 1 of 100
Game 1 ended: 0-1 {Black wins by adjudication}
Hamsters 0.7.1 wins the game as black
Score of XboardEngine vs Hamsters 0.7.1: 0 - 1 - 0

First engine was not identified in the first game, other games are ok.
I did not use the option name, concurrency is set to 1.

This probem did not happen on previous verions 0.2.2.
Thanks for the report. I think I know why this happens now and didn't happen with v0.2.2. Is it possible that your engine sends feature done=1 before myname=engine_name?
No. It is myname first before feature done=1.

But forget about this, sorry only happened if i enable egbb use, both versions actually failed. I initialized egbb before entering xboard mode.
I will check further on this.

micron
Posts: 155
Joined: Mon Feb 15, 2010 8:33 am
Location: New Zealand

Re: Cutechess-cli 0.3.0 released

Post by micron » Sun Oct 10, 2010 3:00 am

I tried the new option

Code: Select all

-srand <n>&#58;&#58;
	Set the random seed for the book move selector to <n>.
expecting that it would change the starting positions in a match.
But it has no discernible effect. With -debug I can see that the positions selected from MyGameStarts.pgn are unchanged.

What is -srand meant to do?

Robert P.

User avatar
ilari
Posts: 750
Joined: Mon Mar 27, 2006 5:45 pm
Location: Finland
Contact:

Re: Cutechess-cli 0.3.0 released

Post by ilari » Sun Oct 10, 2010 6:15 am

micron wrote:I tried the new option

Code: Select all

-srand <n>&#58;&#58;
	Set the random seed for the book move selector to <n>.
expecting that it would change the starting positions in a match.
But it has no discernible effect. With -debug I can see that the positions selected from MyGameStarts.pgn are unchanged.

What is -srand meant to do?

Robert P.
It sets the seed for the random number generator that is used for selecting moves from a Polyglot book file. It has no effect if you use a PGN file as an opening book.

User avatar
WinPooh
Posts: 243
Joined: Fri Mar 17, 2006 7:01 am
Location: Russia
Full name: Vladimir Medvedev
Contact:

Re: Cutechess-cli 0.3.0 released

Post by WinPooh » Tue Nov 09, 2010 8:30 am

When I try to run Aristarch 4.50 under cutechess-cli, I get the message:
Aristarch" doesn't support variant "standard"
This is Aristarch's entry in my engines.json:

Code: Select all

   &#123;
      "command"&#58;"Aristarch.exe",
      "name"&#58;"Aristarch",
      "protocol"&#58;"xboard",
      "workingDirectory"&#58;"engines\\Aristarch"
   &#125;
And this is bat-file I use to run the match:

Code: Select all

set FCP="Fruit"
set SCP="Aristarch"
set TC=5+0.1
set GAMES=60

cutechess-cli.exe -games %GAMES% -repeat -fcp conf=%FCP% -scp conf=%SCP% -both tc=%TC% book=varied.bin bookdepth=7 -pgnout test.pgn min
I am pretty sure Aristarch is capable to play standard chess :)
What do I do wrong?

User avatar
WinPooh
Posts: 243
Joined: Fri Mar 17, 2006 7:01 am
Location: Russia
Full name: Vladimir Medvedev
Contact:

Re: Cutechess-cli 0.3.0 released

Post by WinPooh » Tue Nov 09, 2010 8:50 am

However, when I changed protocol to UCI, it looks ok.

Post Reply