Cutechess-cli 0.4.0 released

Discussion of anything and everything relating to chess playing software and machines.

Moderators: hgm, Rebel, chrisw

User avatar
ilari
Posts: 750
Joined: Mon Mar 27, 2006 7:45 pm
Location: Finland

Cutechess-cli 0.4.0 released

Post by ilari »

It's been almost a year since the previous release, so it's time for another one. Most of the work has been cleanup, optimization, documentation, redesign, etc. stuff that's not visible to the user. But there are also some long-awaited new features. Testing may not have been as thorough as it usually is because I plan on releasing another version pretty soon. Once again I'm interested in hearing about any problems, bugs, feature requests, etc. Unless it's something confidential I recommend posting any feedback here, rather than using private messages. That way other people may be able to help you too.

Like always, I've built versions for 32-bit Windows, 32-bit Linux and 64-bit Linux. Unfortunately I upgraded my Macintosh to OSX Lion, which is not supported by the latest stable version of the Qt framework. The Lion-compatible Qt 4.8 will probably be released soon (the release candidate is out already), so when that happens I'll build a Mac version of cutechess-cli.


What's new in v0.4.0?

* The QJson dependency is gone, we now have our own JSON implementation.
This makes compiling cutechess-cli easier

* Support for multiline engine initialization strings. The lines should be
separated by the \n escape sequence.

* All engine input and output is now captured with the -debug option.
Previously the first few lines of output were often left out.

* Stricter parsing for custom engine options.

* Optimized parsing of engine output. Parsing ouput from chess engines,
especially "info" strings in UCI mode, is by far the biggest performance
bottleneck. In fast games these optimizations can often cut cutechess-cli's
CPU usage in half.

* The engine command and it's command line arguments can be passed in the
same "cmd=" parameter, eg. 'cmd="engine.exe -xbmode"'. If quote characters
are needed, the \" escape sequence should be used.
Example: cmd="\"engine name with spaces.exe\" arg1 arg2"

* Properly trim trailing line breaks from engine output.

* Fixed time per move time controls. There's a new "st=<n>" engine option for
setting the time limit to <n> seconds per move.

* Time margin for the expiry of thinking time. There's a new "timemargin=<n>"
option for allowing engines to go at most <n> milliseconds over the time
limit. This is very useful in "fixed time per move" games which most engines
can't handle without going slightly over the limit.

* Infinite time control. An engine can be given infinite thinking time by
setting the time control parameter to "inf" ("tc=inf"). This can be useful
if it's used in combination with depth and node limits.

* Improved PGN parsing. The parser now copes better with non-standard or
corrupt PGN games.

* Time delay bugfix. Now there shouldn't be any extra games in matches that
use the -wait option.

* Display more information about the match score. After each game
cutechess-cli displays the percentage of points won by the first player,
and the total number of games played.

* Engine options can be set in the engines.json config file. The contents
should look like this:

Code: Select all

&#91;
	&#123;
		"command" &#58; "fruit",
		"name" &#58; "Fruit-2.1",
		"options" &#58; &#91;
			&#123;
				"name" &#58; "Hash",
				"value" &#58; 32
			&#125;,
			&#123;
				"name" &#58; "OwnBook",
				"value" &#58; false
			&#125;,
			&#123;
				"name" &#58; "NullMove Pruning",
				"value" &#58; "Fail High"
			&#125;
		&#93;,
		"protocol" &#58; "uci",
		"workingDirectory" &#58; ""
	&#125;,
	&#123;
		"command" &#58; "crafty",
		"name" &#58; "Crafty-22.10",
		"protocol" &#58; "xboard",
		"whitepov" &#58; true,
		"workingDirectory" &#58; "",
		"initStrings" &#58; &#91;
			"command 1",
			"command 2"
		&#93;
	&#125;
&#93;
Supported XBoard options are "memory", "cores" and "egtpath [TYPE]". And as
usual, options can be set and overridden from the command line with the
"option=value" parameter.


Source code: http://repo.or.cz/w/sloppygui.git
Linux 32-bit binary: http://koti.mbnet.fi/~ilaripih/bin/cute ... x32.tar.gz
Linux 64-bit binary: http://koti.mbnet.fi/~ilaripih/bin/cute ... x64.tar.gz
Windows 32-bit binary: http://koti.mbnet.fi/~ilaripih/bin/cute ... -win32.zip

Thanks for using cutechess-cli!
User avatar
Houdini
Posts: 1471
Joined: Tue Mar 16, 2010 12:00 am

Re: Cutechess-cli 0.4.0 released

Post by Houdini »

Awesome, many thanks!

Robert
Ferdy
Posts: 4833
Joined: Sun Aug 10, 2008 3:15 pm
Location: Philippines

Re: Cutechess-cli 0.4.0 released

Post by Ferdy »

Hi, thanks for this release.
I have tried this version and get a number of loses on time which I could not get from the previous version cutechess-cli 0.3.0. The version 0.3.0 is able to complete matches for a total of 1200 games without problem.

Typical batch file, run on i7-2600k, win7 64 bit os.

Code: Select all

cutechess-cli -fcp
cmd=deuterium.exe dir=C&#58;\myfiles\Chess\engines\nobook\Deuterium0 proto=xboard -s
cp cmd=n2.exe dir=C&#58;\myfiles\Chess\engines\nobook\n2 proto=uci option.Hash=32 -b
oth tc=40/10 -games 100 -repeat -event top50 -pgnin .\opn\top50.pgn -pgnout .\ga
mes\g-i7-top50-low-tc-40movin10sec-exp.pgn -wait 5 -concurrency 3 -resign 5 900
-draw 80 1
One such situation was this against N2. I have not enabled the debug, just captured these from the screen.
...
Started game 6 of 100
Game 4 ended: 1/2-1/2 {Draw by adjudication}
Score of Deuterium v11.02.29.57-2 outpost_1 vs N2 0.4: 1 - 1 - 2 [0.50] 4
Started game 7 of 100
Game 5 ended: 0-1 {Black wins by adjudication}
N2 0.4 wins the game as black
Score of Deuterium v11.02.29.57-2 outpost_1 vs N2 0.4: 1 - 2 - 2 [0.40] 5
Started game 8 of 100
Engine "N2 0.4" failed to respond to ping
Game 6 ended: 0-1 {White's connection stalls}
Deuterium v11.02.29.57-2 outpost_1 wins the game as black
Score of Deuterium v11.02.29.57-2 outpost_1 vs N2 0.4: 2 - 2 - 2 [0.50] 6
ELO difference: 0
Game 7 ended: * {No result}
Score of Deuterium v11.02.29.57-2 outpost_1 vs N2 0.4: 2 - 2 - 2 [0.50] 6
ELO difference: 0
Game 8 ended: * {No result}
Score of Deuterium v11.02.29.57-2 outpost_1 vs N2 0.4: 2 - 2 - 2 [0.50] 6
ELO difference: 0
Warning: QObject::killTimers: timers cannot be stopped from another thread
Finished match
Then against Daydreamer, time loses again with match terminated without finishing.
...
Started game 77 of 100
Game 75 ended: 0-1 {Black wins by adjudication}
Daydreamer 1.75 JA wins the game as black
Score of Deuterium v11.02.29.57-2 outpost_1 vs Daydreamer 1.75 JA: 27 - 34 - 14
[0.45] 75
Started game 78 of 100
Game 77 ended: 0-1 {White loses on time}
Daydreamer 1.75 JA wins the game as black
Score of Deuterium v11.02.29.57-2 outpost_1 vs Daydreamer 1.75 JA: 27 - 35 - 14
[0.45] 76
Started game 79 of 100
Game 78 ended: 0-1 {White's connection stalls}
Deuterium v11.02.29.57-2 outpost_1 wins the game as black
Score of Deuterium v11.02.29.57-2 outpost_1 vs Daydreamer 1.75 JA: 28 - 35 - 14
[0.45] 77
ELO difference: -32
Game 76 ended: * {No result}
Score of Deuterium v11.02.29.57-2 outpost_1 vs Daydreamer 1.75 JA: 28 - 35 - 14
[0.45] 77
ELO difference: -32
Game 79 ended: * {No result}
Score of Deuterium v11.02.29.57-2 outpost_1 vs Daydreamer 1.75 JA: 28 - 35 - 14
[0.45] 77
ELO difference: -32
Finished match
Then against Crafty, Crafty is very stable when it comes to fast time control but also suffers.
Started game 1 of 100
Started game 2 of 100
Started game 3 of 100
Game 2 ended: 1-0 {White wins by adjudication}
Crafty-23.4 PS wins the game as white
Score of Deuterium v11.02.29.57-2 outpost_1 vs Crafty-23.4 PS: 0 - 1 - 0 [0.00]
1
Started game 4 of 100
Game 1 ended: 1/2-1/2 {Draw by insufficient mating material}
Score of Deuterium v11.02.29.57-2 outpost_1 vs Crafty-23.4 PS: 0 - 1 - 1 [0.25]
2
Started game 5 of 100
Game 3 ended: 0-1 {Black wins by adjudication}
Crafty-23.4 PS wins the game as black
Score of Deuterium v11.02.29.57-2 outpost_1 vs Crafty-23.4 PS: 0 - 2 - 1 [0.17]
3
Started game 6 of 100
Game 5 ended: 0-1 {White loses on time}
Crafty-23.4 PS wins the game as black
Score of Deuterium v11.02.29.57-2 outpost_1 vs Crafty-23.4 PS: 0 - 3 - 1 [0.12]
4
Started game 7 of 100
Game 6 ended: 0-1 {White's connection stalls}
Deuterium v11.02.29.57-2 outpost_1 wins the game as black
Score of Deuterium v11.02.29.57-2 outpost_1 vs Crafty-23.4 PS: 1 - 3 - 1 [0.30]
5
ELO difference: -147
Game 4 ended: * {No result}
Score of Deuterium v11.02.29.57-2 outpost_1 vs Crafty-23.4 PS: 1 - 3 - 1 [0.30]
5
ELO difference: -147
Game 7 ended: * {No result}
Score of Deuterium v11.02.29.57-2 outpost_1 vs Crafty-23.4 PS: 1 - 3 - 1 [0.30]
5
ELO difference: -147
Finished match
bhlangonijr
Posts: 482
Joined: Thu Oct 16, 2008 4:23 am
Location: Milky Way

Re: Cutechess-cli 0.4.0 released

Post by bhlangonijr »

Great news!

Thanks for such useful tool.

Regards,
User avatar
ilari
Posts: 750
Joined: Mon Mar 27, 2006 7:45 pm
Location: Finland

Re: Cutechess-cli 0.4.0 released

Post by ilari »

Ferdy wrote:Hi, thanks for this release.
I have tried this version and get a number of loses on time which I could not get from the previous version cutechess-cli 0.3.0. The version 0.3.0 is able to complete matches for a total of 1200 games without problem.

Typical batch file, run on i7-2600k, win7 64 bit os.

Code: Select all

cutechess-cli -fcp
cmd=deuterium.exe dir=C&#58;\myfiles\Chess\engines\nobook\Deuterium0 proto=xboard -s
cp cmd=n2.exe dir=C&#58;\myfiles\Chess\engines\nobook\n2 proto=uci option.Hash=32 -b
oth tc=40/10 -games 100 -repeat -event top50 -pgnin .\opn\top50.pgn -pgnout .\ga
mes\g-i7-top50-low-tc-40movin10sec-exp.pgn -wait 5 -concurrency 3 -resign 5 900
-draw 80 1
One such situation was this against N2. I have not enabled the debug, just captured these from the screen.
I tried the same setup in Linux, ran a couple of 100 game matches, and got 3 time forfeits in total, all from N2. So nothing abnormal there. It seems that this problem is with the Windows version, especially QObject::killTimers warning hints that way. I'll see if I can reproduce this in Windows 7...
Ferdy
Posts: 4833
Joined: Sun Aug 10, 2008 3:15 pm
Location: Philippines

Re: Cutechess-cli 0.4.0 released

Post by Ferdy »

ilari wrote:
Ferdy wrote:Hi, thanks for this release.
I have tried this version and get a number of loses on time which I could not get from the previous version cutechess-cli 0.3.0. The version 0.3.0 is able to complete matches for a total of 1200 games without problem.

Typical batch file, run on i7-2600k, win7 64 bit os.

Code: Select all

cutechess-cli -fcp
cmd=deuterium.exe dir=C&#58;\myfiles\Chess\engines\nobook\Deuterium0 proto=xboard -s
cp cmd=n2.exe dir=C&#58;\myfiles\Chess\engines\nobook\n2 proto=uci option.Hash=32 -b
oth tc=40/10 -games 100 -repeat -event top50 -pgnin .\opn\top50.pgn -pgnout .\ga
mes\g-i7-top50-low-tc-40movin10sec-exp.pgn -wait 5 -concurrency 3 -resign 5 900
-draw 80 1
One such situation was this against N2. I have not enabled the debug, just captured these from the screen.
I tried the same setup in Linux, ran a couple of 100 game matches, and got 3 time forfeits in total, all from N2. So nothing abnormal there. It seems that this problem is with the Windows version, especially QObject::killTimers warning hints that way. I'll see if I can reproduce this in Windows 7...
Here is another situation Deuterium vs Daydreamer, I test this with -debug to see the details and set concurrency to 1.
What happened was Deuterium has just finished searching and cutechess-cli issued ? command then Deuterium send the move, but was flagged as lost on time, although there was still plenty of time. See log below.

Code: Select all

285217 >Deuterium v11.02.29.57-2 outpost_1&#40;0&#41;&#58; time 568
otim 286
285217 >Deuterium v11.02.29.57-2 outpost_1&#40;0&#41;&#58; c3e4
285217 <Deuterium v11.02.29.57-2 outpost_1&#40;0&#41;&#58; # TimeLeft=5680.00 ms.  MaxTime=284.00 ms  AllocatedTime=265.00 ms.
285217 <Deuterium v11.02.29.57-2 outpost_1&#40;0&#41;&#58; # ply score  time     nodes Pv line
285217 <Deuterium v11.02.29.57-2 outpost_1&#40;0&#41;&#58; # -----------------------------------
285217 <Deuterium v11.02.29.57-2 outpost_1&#40;0&#41;&#58;   1    76      0         40 b7b5
285217 <Deuterium v11.02.29.57-2 outpost_1&#40;0&#41;&#58;   2    57      0        213 h2h3 e4c5 h3g3 g1f2
285217 <Deuterium v11.02.29.57-2 outpost_1&#40;0&#41;&#58;   3    57      0        493 h2h3 e4c5 h3g3 g1f2
285217 <Deuterium v11.02.29.57-2 outpost_1&#40;0&#41;&#58;   4    57      0       1297 h2h3 e4c5 a8a7 f1f2
285217 <Deuterium v11.02.29.57-2 outpost_1&#40;0&#41;&#58;   5    43      0       3187 h2h3 g4g5 h8d8 f3e2 d8d5
285233 <Deuterium v11.02.29.57-2 outpost_1&#40;0&#41;&#58;   6    26      1       5584 h2h3 g4g5 h8d8 f3g2 h4g4 f1f7
285233 <Deuterium v11.02.29.57-2 outpost_1&#40;0&#41;&#58;   7    56      1       9626 h2h3 &#40;1.0&#41;
285233 <Deuterium v11.02.29.57-2 outpost_1&#40;0&#41;&#58;   7    22      1      14823 h2h3 g4g5 h4h7 e4c5 h3h2 b4b5 a6b5 &#40;1.0&#41;
285248 <Deuterium v11.02.29.57-2 outpost_1&#40;0&#41;&#58;   8    20      3      27343 h2h3 g4g5 h4h7 e4c5 h3h2 f3e4 h7h5 f1f8 h8f8 e5h2 h5g5 g1h1 &#40;1.9&#41;
285264 <Deuterium v11.02.29.57-2 outpost_1&#40;0&#41;&#58;   9     0      4      49152 h2h3 g4g5 h4h7 e4c5 h7h4 c5e4 &#40;1.5&#41;
285342 <Deuterium v11.02.29.57-2 outpost_1&#40;0&#41;&#58;  10     0     12     124765 h2h3 g4g5 h8f8 g1g2 f8h8 g2g1 &#40;2.7&#41;
285420 <Deuterium v11.02.29.57-2 outpost_1&#40;0&#41;&#58;  11     0     20     212595 h2h3 g4g5 h8f8 g1g2 f8h8 g2g1 &#40;1.6&#41;
285420 <Deuterium v11.02.29.57-2 outpost_1&#40;0&#41;&#58; # Tm=203 ms, NPS=1047K, Mv=h2h3, Val=0, Hply=124, Fifty=4
291114 >Deuterium v11.02.29.57-2 outpost_1&#40;0&#41;&#58; ?
291114 <Deuterium v11.02.29.57-2 outpost_1&#40;0&#41;&#58; move h2h3
291114 >Daydreamer 1.75 JA&#40;1&#41;&#58; isready
291114 >Deuterium v11.02.29.57-2 outpost_1&#40;0&#41;&#58; force
291114 >Deuterium v11.02.29.57-2 outpost_1&#40;0&#41;&#58; result 1-0 &#123;Black loses on time&#125;
291114 >Deuterium v11.02.29.57-2 outpost_1&#40;0&#41;&#58; ping 31
291114 <Deuterium v11.02.29.57-2 outpost_1&#40;0&#41;&#58; Error &#40;unknown command&#41;&#58; ?
291114 <Daydreamer 1.75 JA&#40;1&#41;&#58; readyok
291114 <Deuterium v11.02.29.57-2 outpost_1&#40;0&#41;&#58; pong 31
Game 8 ended&#58; 1-0 &#123;Black loses on time&#125;
Daydreamer 1.75 JA wins the game as white
Score of Deuterium v11.02.29.57-2 outpost_1 vs Daydreamer 1.75 JA&#58; 2 - 4 - 2  &#91;0.38&#93; 8
Started game 9 of 100
User avatar
ilari
Posts: 750
Joined: Mon Mar 27, 2006 7:45 pm
Location: Finland

Re: Cutechess-cli 0.4.0 released

Post by ilari »

Okay, I managed to reproduce the ping timeout problem in Windows, and I think I've just solved it. I'll run some more games to be sure, and then post the fix.

Here is another situation Deuterium vs Daydreamer, I test this with -debug to see the details and set concurrency to 1.
What happened was Deuterium has just finished searching and cutechess-cli issued ? command then Deuterium send the move, but was flagged as lost on time, although there was still plenty of time. See log below.
Actually, judging by the time stamps it seems that Deuterium got its turn at 285217 ms, and sent its move at 291114 ms, which is a move time of 5897 ms, that is 217 ms over the limit. So the time forfeit was justified. Was Deuterium supposed to send its move immediately after this line?:

Code: Select all

# Tm=203 ms, NPS=1047K, Mv=h2h3, Val=0, Hply=124, Fifty=4
User avatar
ilari
Posts: 750
Joined: Mon Mar 27, 2006 7:45 pm
Location: Finland

Re: Cutechess-cli 0.4.0 released

Post by ilari »

ilari wrote:Was Deuterium supposed to send its move immediately after this line?:

Code: Select all

# Tm=203 ms, NPS=1047K, Mv=h2h3, Val=0, Hply=124, Fifty=4
Now that I've seen some more of Deuterium's debug output, then answer is clearly yes, which means cutechess-cli lost that line. A fix is coming.
Ferdy
Posts: 4833
Joined: Sun Aug 10, 2008 3:15 pm
Location: Philippines

Re: Cutechess-cli 0.4.0 released

Post by Ferdy »

ilari wrote:
ilari wrote:Was Deuterium supposed to send its move immediately after this line?:

Code: Select all

# Tm=203 ms, NPS=1047K, Mv=h2h3, Val=0, Hply=124, Fifty=4
Now that I've seen some more of Deuterium's debug output, then answer is clearly yes, which means cutechess-cli lost that line. A fix is coming.
Yes it is supposed to send the move after that line. Thanks.
User avatar
lucasart
Posts: 3232
Joined: Mon May 31, 2010 1:29 pm
Full name: lucasart

Re: Cutechess-cli 0.4.0 released

Post by lucasart »

Thanks a lot for cutechess-cli. This program is great and it's the only thing I use now, to test my engine.