IanO wrote:Another problem: in a match between Hamsters and Crafty 20.14, several of the games were lost by crafty due to an illegal first move, which turns out to be a legal move in the preceding drawn game (once by 50-move rule, twice by threefold repetition). Looks like a race condition between starting a new game and waiting for the previous program to quit the previous game.
Thanks for the report. It would be easy to tackle this problem by synchronizing the engine with cutechess-cli at the end of a game with the ping command, but since all engines don't support it, I'm going to have to come up with something better.
As promised, I've released a new version (0.1.1) of Cutechess-cli. Links to the binaries are in the first post. All issues that were reported to me should be fixed, and most of the features requested are there as well:
- On Linux/OSX the startup script is fixed so one can use paths with spaces in the "dir" engine option, as long as they're wrapped in quotes of course.
- It's possible to include the full path to the executable in the "cmd" engine option, even if it contains spaces.
- There's a new "arg" engine option which allows the user to pass arguments to engines. This option can be used multiple times. So to set Crafty's book path one would type: 'cmd=crafty arg="bookpath=/opt/local/share/crafty"'. To run a Windows engine under wine in Linux, use 'cmd=wine arg=engine.exe'.
- Cutechess-cli now makes sure that engines have finished playing before a new game is started. Every engine that supports the ping command shoud work without problems. Of the rest, at least 90% should work, but there could be special cases.
- There are now engine options to set limits to search depth and node count (run cutechess-cli --help to learn more about them). Currently the node limit works only for UCI engines. A time control must still be specified when these limits are used. So if you don't want the time control to affect the search time, just set it to a very big value.
- As suggested by Marc Lacrosse, I added draw and resign adjudication options. A couple of examples:
-- Adjudicate as draw if both engines give a score between -10 and +10 centipawns after 250 fullmoves: "-draw 250 10".
-- Adjudicate as loss when an engine's score is at least 500 centipawns below zero for at least 5 consecutive moves: "-resign 5 500".
I implemented the -resign option this way because passing a negative value would have the "-" sign, and would be considered a separate command line argument.
- There's a "invertscores" engine option that should be used with engines that report evaluation scores from white's point of view. Crafty is the only one I know of. If you use the resign adjudication it's very important that the "invertscores" option is used when applicable.
I tested all the new features and fixes with several engines (Xb1, Xb2, UCI) and on all three supported platforms, but it may not have been extensive enough. So again, I'd appreciate any feedback. I always take bug reports and feature requests seriously.
Thanks Ilari for including the node limit for UCI engines.
How about adding tournament support? For example there will be a text file that has the tournament settings. Then the user can play a tournament and if he chooses to stop the tournament, the current results should be saved and the tournament can be resumed on another time. The tournament settings should easily be editable like the opponents, time control, starting positions, etc.
Edsel Apostol wrote:Thanks Ilari for including the node limit for UCI engines.
No problem, that was an easy one.
How about adding tournament support? For example there will be a text file that has the tournament settings. Then the user can play a tournament and if he chooses to stop the tournament, the current results should be saved and the tournament can be resumed on another time. The tournament settings should easily be editable like the opponents, time control, starting positions, etc.
We are going to support tournaments, but that's going to take more time to do. I've already designed an XML file format for tournaments. The next step is to write the Tournament class, the configuration model and other supporting classes, and put them in the chess library. Then we can use the same tournament code in GUI and CLI side. So it will be possible to set up a tournament in the GUI and then run it with Cutechess-cli.
This looks like a useful tool -- thanks for sharing!
I just gave the Mac OS X version a quick try, and tried running a 1000 game match between two experimental Glaurung versions at 40 moves/1 second (which may seem like a crazy time control, but keep in mind that this program is written to run on a machine 100 times slower than my computer). After 687 games, the program crashed with the following error message:
This looks like a useful tool -- thanks for sharing!
I just gave the Mac OS X version a quick try, and tried running a 1000 game match between two experimental Glaurung versions at 40 moves/1 second (which may seem like a crazy time control, but keep in mind that this program is written to run on a machine 100 times slower than my computer). After 687 games, the program crashed with the following error message:
Started game 688 of 1000
./cutechess-cli.sh: line 13: 78392 Bus error $dirname/$appname "$@"
Tord
Thanks for report. I managed to reproduce the problem with my debug build which aborted on an assertion. The crash happened when one of the engines lost on time.
The problem should be fixed now. I've updated the binaries and commited the fix to our source code repository.
Thanks a bunch for providing this tool, and again for making it open source. It looks to be a very good solution for me for background testing, and will make a good complement to xboard (for live viewing).
I suppose I should release my code at some point, but it is really tied to unix. It is what I use on our cluster and will handle using multiple starting positions, play as many games per position as you want, alternating colors (I use 2 here), and creates a PGN file containing all the games with results. I have another program that will take a bunch of scripts that use the above program, and execute as many at once as you want, so that you can use all your CPUs or just 1/2 of them or whatever. But again, all is pure Unix-based, no windows at all, and documentation is <nil>
Another little bug report. I actually noticed this because I was setting the time control to 1+0, which turns out to be 1 second/game. I think that's rather confusing, as it breaks with the xboard protocol (which would specify one second as 0:01). But anyways, it sends the level command to the engine without 0-padding the time, so the engine receives "level 0 0:1 0". I traced the cause to this function, hoping I could fix it, but I know nothing about Qt.
static QString msToXboardTime(int ms)
{
int sec = ms / 1000;
QString number = QString::number(sec / 60);
if (sec % 60 != 0)
number += QString(":%1").arg(sec % 60);
return number;
}
One other request: could you put an interrupt handler in the program? This is a pretty minor issue, but sometimes I need to kill the program, and the logfile just abruptly ends now. It would be nice to have a clean break there, and maybe save the first part of the game in the pgn file.