Cutechess-cli: A command line tool for engine-engine matches

Discussion of anything and everything relating to chess playing software and machines.

Moderators: hgm, Rebel, chrisw

User avatar
ilari
Posts: 750
Joined: Mon Mar 27, 2006 7:45 pm
Location: Finland

Re: Cutechess-cli: A command line tool for engine-engine mat

Post by ilari »

Dann Corbit wrote:It would be very nice if it could support USI also (there is no tournament GUI for USI at all that I know of, though you can play a single engine against a single engine for N games with Shogidokoro):
http://www.glaurungchess.com/shogi/usi.html
Implementing a new chess protocol wouldn't be that hard, but implementing a new chess variant like Shogi certainly would. We're going to redesign the chess protocol classes a bit so that 3rd party developers can easily add new protocols just by subclassing the ChessEngine class, without modifying the chesslib source code. I'm thinking of doing the same thing for the chessboard and chess variant code. Then it would be possible to add new variants without too much pain.

I'll think about this. The feature set of Cute Chess 1.0 is pretty much frozen, but who knows what happens after that.
User avatar
hgm
Posts: 27787
Joined: Fri Mar 10, 2006 10:06 am
Location: Amsterdam
Full name: H G Muller

Re: Cutechess-cli: A command line tool for engine-engine mat

Post by hgm »

Dann Corbit wrote:It would be very nice if it could support USI also (there is no tournament GUI for USI at all that I know of, though you can play a single engine against a single engine for N games with Shogidokoro):
http://www.glaurungchess.com/shogi/usi.html
Perhaps it would be easier to adapt Polyglot to USI. Then all USI engines could run under WinBoard+PSWBTM, just like all UCCI engines already run under WinBoard.
User avatar
ilari
Posts: 750
Joined: Mon Mar 27, 2006 7:45 pm
Location: Finland

Re: Cutechess-cli 0.1.9 released

Post by ilari »

When there are new features there are new bugs. Version 0.1.9 fixes some bugs that were introduced in 0.1.8:

- Crash due to a player resigning on the opponent's turn
- Crash due to a player disconnecting or being terminated
- Crash/freeze due to failure to respond to ping

Thanks for everyone who submitted bug reports. I've updated the binaries.
Gian-Carlo Pascutto
Posts: 1243
Joined: Sat Dec 13, 2008 7:00 pm

Re: Cutechess-cli 0.1.9 released

Post by Gian-Carlo Pascutto »

Today I got this, first time I ever saw it:

Score of Deep Sjeng 3.2 LE2 vs Deep Sjeng 3.2 BL14: 786 - 885 - 1292
Started game 2967 of 5000
pure virtual method called
terminate called without an active exception
Aborted
User avatar
ilari
Posts: 750
Joined: Mon Mar 27, 2006 7:45 pm
Location: Finland

Re: Cutechess-cli 0.1.9 released

Post by ilari »

Gian-Carlo Pascutto wrote:Today I got this, first time I ever saw it:

Score of Deep Sjeng 3.2 LE2 vs Deep Sjeng 3.2 BL14: 786 - 885 - 1292
Started game 2967 of 5000
pure virtual method called
terminate called without an active exception
Aborted
Whoa, that's interesting. I hate it when that happens at game 3000 or so, makes it pretty annoying to debug. A few questions:
- What cutechess-cli settings/command did you use? I'm most interested in the time control and concurrency settings.
- Which platform (Windows/Mac/Linux, 32-bit or 64-bit)?
- If you're running Linux, which version of Qt do you have?
- Is it one of my builds or did you compile cutechess-cli yourself?
Gian-Carlo Pascutto
Posts: 1243
Joined: Sat Dec 13, 2008 7:00 pm

Re: Cutechess-cli 0.1.9 released

Post by Gian-Carlo Pascutto »

ilari wrote: Whoa, that's interesting. I hate it when that happens at game 3000 or so, makes it pretty annoying to debug. A few questions:
- What cutechess-cli settings/command did you use? I'm most interested in the time control and concurrency settings.
- Which platform (Windows/Mac/Linux, 32-bit or 64-bit)?
- If you're running Linux, which version of Qt do you have?
- Is it one of my builds or did you compile cutechess-cli yourself?
Own compile from your:
commit 178918d6df59ac0c7e7cbae177907187de46e3ae

64 bit AMD64 Linux
gcc (Debian 4.3.4-5) 4.3.4

Package: libqt4-core
Version: 4:4.5.3-2

Command line something like this:
./cutechess-cli -fcp cmd=./sjengns4 dir=~/chess/sjeng/ proto=uci -scp cmd=./sjengbl14 dir=~/chess/sjeng proto=uci -both tc=5+0.1 -book book.bin -draw 80 1 -resign 3 500 -games 5000 -repeat -pgnout ns4bl14.pgn -concurrency 4

Whatever the bug was, it must be very seldom. I've played a few hundred thousand games with this tool already and never hit this before. Some obscure threading bug?
User avatar
ilari
Posts: 750
Joined: Mon Mar 27, 2006 7:45 pm
Location: Finland

Re: Cutechess-cli 0.1.9 released

Post by ilari »

Gian-Carlo Pascutto wrote: Own compile from your:
commit 178918d6df59ac0c7e7cbae177907187de46e3ae

64 bit AMD64 Linux
gcc (Debian 4.3.4-5) 4.3.4

Package: libqt4-core
Version: 4:4.5.3-2

Command line something like this:
./cutechess-cli -fcp cmd=./sjengns4 dir=~/chess/sjeng/ proto=uci -scp cmd=./sjengbl14 dir=~/chess/sjeng proto=uci -both tc=5+0.1 -book book.bin -draw 80 1 -resign 3 500 -games 5000 -repeat -pgnout ns4bl14.pgn -concurrency 4
Looks fine. The Qt version was the biggest thing I was worried about. We only test Cute Chess with Qt 4.5., so previous versions may not work so well.
Whatever the bug was, it must be very seldom. I've played a few hundred thousand games with this tool already and never hit this before. Some obscure threading bug?
Hard to say. I'm pretty sure we're not calling virtual function from a constructor so this could be a threading + dangling pointer problem. GDB would probably show a nice backtrace if we encountered this error during a debugging run.
Gian-Carlo Pascutto
Posts: 1243
Joined: Sat Dec 13, 2008 7:00 pm

Re: Cutechess-cli 0.1.9 released

Post by Gian-Carlo Pascutto »

Maybe it can be reproduced by running with very high concurrencies.
User avatar
ilari
Posts: 750
Joined: Mon Mar 27, 2006 7:45 pm
Location: Finland

Re: Cutechess-cli 0.1.9 released

Post by ilari »

Gian-Carlo Pascutto wrote:Maybe it can be reproduced by running with very high concurrencies.
Maybe. One bug was discovered in the Windows version when running with "-concurrency 4" while "-concurrency 2" worked fine. What may help even more is running the games really quickly, like 1 sec/game + agressive draw and resign adjudications. This error looks like something that would happen between games rather than during them.
Dann Corbit
Posts: 12537
Joined: Wed Mar 08, 2006 8:57 pm
Location: Redmond, WA USA

Re: Millispeconds and a new EPD spec

Post by Dann Corbit »

sje wrote:
ilari wrote:I've uploaded a new version (0.1.3) with a few changes:

- Support millisecond-precision in time controls
I agree that millisecond resolution is the way to go. Piloted spacecraft use a millisecond period update for critical telemetry; if it's good enough for a spaceship then I suppose it's good enough for a chess clock.

A new EPD specification is coming, and it uses a tighter formalism for a number of things including time values. Example:

Code: Select all

r1bqkbnr/pppp1ppp/2n5/8/3Q4/2N5/PPP1PPPP/R1B1KBNR w KQkq - 1 4 acd 6; acn 1436575; cc 4:55.265 4:54.606; pes +0.020; pv Qd3 h6 Nf3 Nf6 e4 Bd6; te 1.445; tu 1.333;
Note:

1) The half move counter and the full move number now appear explicitly as the fifth and sixth fields of a standard FEN position specification. The earlier EPD version had these omitted or appearing as optional operations (hmvc and fmvn).

2) Most things don't change: acd, acn, am, bm, pv, sv, etc.

3) Some things are deprecated, such as pm (use the first parameter value of pv instead).
I find pm useful (predicted move may not be the same thing as best move for instance). I have code that uses the pm field.
4) All time values share the same the same format of DDD:HH:MM:SS.mmm with optional leading zero suppression. Example: cc (chess clock) has a White and a Black countdown time operand.

5) Instead of acs, te records analysis elapsed (wall) time, and tu represents usage time. Usage may be greater than elapsed time for multicore analysis.

6) The ce operator is deprecated; the replacement pes (pawn equivalent score) operator gives an evaluation in signed floating point pawns with millipawn resolution. There are a few non-decimal values: MateInN (e.g., MateIn3), LoseInN, Even, PosInf, NegInf, and Broken.
Perhaps inconvenient for some, since almost nobody uses floating point for evaluation. But if you are going to make it floating point, and 1.0 is supposed to represent a pawn, then why use milliseconds? Just let it be a double and then they can have an internal precision that they like. In any case, we're going to use sscanf("%lf"... to collect it.
7) A few operations are undecided at the moment, like dm and solve.
I find dm very useful. I do not even know what solve is for because I have never run across that field. People are going to use different scales for checkmate (even though you have 32767 listed as written in stone, lots of people use 9999, 99999 or some other value instead, so dm has a clear purpose here).

8) Symbol operands are never quoted and string operands are always quoted. Examples:

id WAC.241;
I have literally hundreds of thousands of id fields with spaces in them. There was nothing in the original spec to prevent that and I will find it most annoying if I am not allowed to quote id fields. In fact, I can tell you now that I will ignore that new part of the spec because I have not enough time nor the compulsion to correct them.

name "Win at Chess problem 241";