Cutechess-cli: A command line tool for engine-engine matches

[Werner]

Hello Ilari,

Thanks a bunch for providing this tool, and again for making it open source. It looks to be a very good solution for me for background testing, and will make a good complement to xboard (for live viewing).

Thanks. I needed this tool myself for testing the core functionality of our GUI, and for testing Sloppy. I've found it to work fine even for lightning-fast games (eg. 1 second per game), which Xboard/Winboard can't handle that well.

I cannot play such ultrafast games. Even with 40 moves in 20 sec the engine looses sometimes on time. Is this a fault of the engine or do I handle something false?

my start.bat is:

cutechess-cli -fcp cmd=naum.exe proto=xboard -scp cmd=fruit235m-64 proto=uci uci/Hash=64 uci/NalimovPath=J:\tbs -both tc=40/20 -games 2 -pgnin top25.pgn -repeat -pgnout games.pgn

[Event "?"]
[Site "?"]
[Date "2009.04.11"]
[Round "2"]
[White "Fruit 2.3.5m Test Beta"]
[Black "Naum 4"]
[Result "0-1"]
[PlyCount "152"]
[TimeControl "40/20"]

1. c4 {book} e5 {book} 2. Nc3 {book} Nf6 {book} 3. Nf3 {book} Nc6 {book}
4. g3 {book} Bb4 {book} 5. Bg2 {book} O-O {book} 6. O-O {book} e4 {book}
7. Ng5 {+0.18/10 1s} Ng4 {-0.24/9 0s} 8. Ngxe4 {+0.39/10 1s} f5 {-0.34/10 0s}
9. a3 {+0.27/11 1s} Be7 {-0.07/9 0s} 10. f3 {+0.67/11 1s} Nge5 {-0.41/10 0s}
11. f4 {+0.61/11 1s} Nxc4 {-0.51/10 0s} 12. Ng5 {+0.66/11 2s} Kh8 {-0.42/11 1s}
13. Nd5 {+0.75/10 0s} Bf6 {-0.40/10 1s} 14. e3 {+0.94/9 1s} Bxg5 {-0.83/10 0s}
15. fxg5 {+1.25/9 1s} d6 {-0.84/11 1s} 16. Qh5 {+1.13/10 1s} N4e5 {-0.98/10 1s}
17. d4 {+1.41/11 1s} Ng6 {-0.81/10 0s} 18. Bd2 {+1.35/10 1s} Be6 {-0.91/11 1s}
19. Rac1 {+0.99/9 0s} Qd7 {-0.81/10 1s} 20. Qe2 {+0.97/9 1s} Bxd5 {-0.60/11 0s}
21. Bxd5 {+0.86/10 1s} Nge7 {-0.57/12 1s} 22. Bc4 {+0.87/10 1s} a5 {-0.54/11 0s}
23. Qh5 {+0.85/10 1s} d5 {-0.50/12 0s} 24. Bd3 {+0.79/10 1s} Qe6 {-0.59/11 1s}
25. Rf2 {+0.83/9 1s} Rf7 {-0.50/10 0s} 26. Qh4 {+0.86/8 0s} Kg8 {-0.51/12 1s}
27. Rcf1 {+0.74/9 1s} Raf8 {-0.54/11 1s} 28. Qh5 {+0.75/9 0s} Rd8 {-0.56/11 1s}
29. Bc2 {+0.83/9 1s} Rdf8 {-0.46/10 0s} 30. Qh3 {+0.72/9 1s} Qd7 {-0.50/11 0s}
31. Rf4 {+0.78/9 0s} Qe6 {-0.45/10 1s} 32. Bd3 {+0.73/10 0s} Qg6 {-0.49/11 0s}
33. Qh4 {+0.74/10 0s} Nd8 {-0.19/10 0s} 34. Bxa5 {+0.40/8 0s} Ne6 {-0.12/11 0s}
35. R4f2 {+0.32/9 0s} Nc6 {0.00/11 0s} 36. Bd2 {+0.37/9 0s} Nxg5 {+0.11/10 1s}
37. Rf4 {0.00/8 0s} h6 {+0.04/11 1s} 38. a4 {+0.46/1 0s} Ne4 {+0.10/11 1s}
39. Bxe4 {+0.35/1 0s} dxe4 {+0.10/11 0s} 40. Bc3 {+0.38/1 0s} Qe6 {+0.16/9 0s}
41. Qh3 {-0.06/10 1s} Ne7 {+0.23/11 0s} 42. Bb4 {-0.42/11 1s} Qb3 {+0.63/10 0s}
43. Bxe7 {-0.56/12 1s} Qxe3+ {+0.55/10 0s} 44. Kh1 {-0.72/13 1s}
Rxe7 {+0.75/11 0s} 45. Rxf5 {-0.68/12 1s} Rfe8 {+0.83/10 0s}
46. Rd5 {-0.85/11 1s} Qb3 {+1.07/11 1s} 47. Rd7 {-0.91/10 0s} Qxa4 {+1.00/11 0s}
48. Rxe7 {-0.88/11 1s} Rxe7 {+1.00/6 0s} 49. Qc8+ {-0.98/10 1s}
Kh7 {+1.05/12 0s} 50. Qf5+ {-1.22/11 1s} g6 {+1.20/13 1s} 51. Qd5 {-1.24/12 0s}
Kg7 {+1.22/12 0s} 52. Qxb7 {-1.06/10 1s} Qxd4 {+1.32/11 1s}
53. Qa8 {-1.18/10 1s} Qd7 {+1.30/11 0s} 54. Rf8 {-0.61/7 0s} Qe6 {+1.51/9 0s}
55. Rf4 {-1.13/8 1s} e3 {+2.14/10 0s} 56. Rf1 {-2.79/11 2s} e2 {+2.42/10 0s}
57. Re1 {-3.58/12 1s} Qf6 {+2.42/12 0s} 58. Qg2 {-2.66/11 0s} Qxb2 {+2.82/11 0s}
59. Qf2 {-2.77/10 0s} Qe5 {+3.00/11 1s} 60. Kg2 {-3.06/11 0s} c5 {+3.01/11 0s}
61. Qf4 {-2.93/12 0s} Qd4 {+3.01/11 0s} 62. h4 {-3.06/10 1s} Qc3 {+2.93/11 1s}
63. Qf2 {-3.06/9 0s} Qe5 {+2.86/13 1s} 64. Qf4 {-3.40/11 0s} Qd5+ {+2.86/12 1s}
65. Kg1 {-3.33/9 0s} c4 {+3.52/10 1s} 66. Kh2 {-3.74/6 0s} Qc5 {+3.91/11 0s}
67. Qc1 {-4.53/7 0s} c3 {+4.06/11 1s} 68. Kg2 {-4.81/9 0s} Re3 {+5.28/10 1s}
69. Kf2 {-4.77/8 0s} Rd3+ {+10.45/11 1s} 70. Kg2 {-7.36/10 0s}
Qd5+ {+11.94/11 1s} 71. Kh2 {-10.62/12 1s} Qe5 {+11.07/13 0s}
72. Rg1 {-8.84/6 0s} Rxg3 {+11.98/15 1s} 73. Rxg3 {-7.42/1 0s}
e1=Q {+11.98/15 0s} 74. Qxe1 {-7.42/1 0s} Qxe1 {+11.98/16 1s}
75. Rg4 {-8.14/1 0s} c2 {+11.98/12 0s} 76. Rg1 {-11.87/1 0s}
Qxh4+ {+22.17/12 0s} {White loses on time} 0-1

[ilari]

I don't think the windows API is that bad, since crafty already works with windows threads and such. But this was a one-off project for my cluster testing and I had no intention of trying to distribute or support it outside of my lab here. Which meant a quick-and-dirty unix implementatoin, which is a system I know inside-out.

As someone who's used POSIX threads a lot, I just can't stand the lack of condition variables in Windows. And I absolutely hate the WinAPI documentation, naming conventions and some of the data types (PHANDLE, etc.).

[ilari]

Hello Ilari,

Thanks a bunch for providing this tool, and again for making it open source. It looks to be a very good solution for me for background testing, and will make a good complement to xboard (for live viewing).

Thanks. I needed this tool myself for testing the core functionality of our GUI, and for testing Sloppy. I've found it to work fine even for lightning-fast games (eg. 1 second per game), which Xboard/Winboard can't handle that well.

I cannot play such ultrafast games. Even with 40 moves in 20 sec the engine looses sometimes on time. Is this a fault of the engine or do I handle something false?

It's not the engine's fault. Currently ultrafast games work only on Linux and Mac OS. See this post for an explanation: http://www.talkchess.com/forum/viewtopi ... 452#258452

I'm working on this problem, but it may take a week or two to fix because I'll have to get dirty with WinAPI pipes to fix it.

[Edsel Apostol]

Hello Ilari,

Thanks a bunch for providing this tool, and again for making it open source. It looks to be a very good solution for me for background testing, and will make a good complement to xboard (for live viewing).

Thanks. I needed this tool myself for testing the core functionality of our GUI, and for testing Sloppy. I've found it to work fine even for lightning-fast games (eg. 1 second per game), which Xboard/Winboard can't handle that well.

I cannot play such ultrafast games. Even with 40 moves in 20 sec the engine looses sometimes on time. Is this a fault of the engine or do I handle something false?

my start.bat is:

cutechess-cli -fcp cmd=naum.exe proto=xboard -scp cmd=fruit235m-64 proto=uci uci/Hash=64 uci/NalimovPath=J:\tbs -both tc=40/20 -games 2 -pgnin top25.pgn -repeat -pgnout games.pgn

[Event "?"]
[Site "?"]
[Date "2009.04.11"]
[Round "2"]
[White "Fruit 2.3.5m Test Beta"]
[Black "Naum 4"]
[Result "0-1"]
[PlyCount "152"]
[TimeControl "40/20"]

1. c4 {book} e5 {book} 2. Nc3 {book} Nf6 {book} 3. Nf3 {book} Nc6 {book}
4. g3 {book} Bb4 {book} 5. Bg2 {book} O-O {book} 6. O-O {book} e4 {book}
7. Ng5 {+0.18/10 1s} Ng4 {-0.24/9 0s} 8. Ngxe4 {+0.39/10 1s} f5 {-0.34/10 0s}
9. a3 {+0.27/11 1s} Be7 {-0.07/9 0s} 10. f3 {+0.67/11 1s} Nge5 {-0.41/10 0s}
11. f4 {+0.61/11 1s} Nxc4 {-0.51/10 0s} 12. Ng5 {+0.66/11 2s} Kh8 {-0.42/11 1s}
13. Nd5 {+0.75/10 0s} Bf6 {-0.40/10 1s} 14. e3 {+0.94/9 1s} Bxg5 {-0.83/10 0s}
15. fxg5 {+1.25/9 1s} d6 {-0.84/11 1s} 16. Qh5 {+1.13/10 1s} N4e5 {-0.98/10 1s}
17. d4 {+1.41/11 1s} Ng6 {-0.81/10 0s} 18. Bd2 {+1.35/10 1s} Be6 {-0.91/11 1s}
19. Rac1 {+0.99/9 0s} Qd7 {-0.81/10 1s} 20. Qe2 {+0.97/9 1s} Bxd5 {-0.60/11 0s}
21. Bxd5 {+0.86/10 1s} Nge7 {-0.57/12 1s} 22. Bc4 {+0.87/10 1s} a5 {-0.54/11 0s}
23. Qh5 {+0.85/10 1s} d5 {-0.50/12 0s} 24. Bd3 {+0.79/10 1s} Qe6 {-0.59/11 1s}
25. Rf2 {+0.83/9 1s} Rf7 {-0.50/10 0s} 26. Qh4 {+0.86/8 0s} Kg8 {-0.51/12 1s}
27. Rcf1 {+0.74/9 1s} Raf8 {-0.54/11 1s} 28. Qh5 {+0.75/9 0s} Rd8 {-0.56/11 1s}
29. Bc2 {+0.83/9 1s} Rdf8 {-0.46/10 0s} 30. Qh3 {+0.72/9 1s} Qd7 {-0.50/11 0s}
31. Rf4 {+0.78/9 0s} Qe6 {-0.45/10 1s} 32. Bd3 {+0.73/10 0s} Qg6 {-0.49/11 0s}
33. Qh4 {+0.74/10 0s} Nd8 {-0.19/10 0s} 34. Bxa5 {+0.40/8 0s} Ne6 {-0.12/11 0s}
35. R4f2 {+0.32/9 0s} Nc6 {0.00/11 0s} 36. Bd2 {+0.37/9 0s} Nxg5 {+0.11/10 1s}
37. Rf4 {0.00/8 0s} h6 {+0.04/11 1s} 38. a4 {+0.46/1 0s} Ne4 {+0.10/11 1s}
39. Bxe4 {+0.35/1 0s} dxe4 {+0.10/11 0s} 40. Bc3 {+0.38/1 0s} Qe6 {+0.16/9 0s}
41. Qh3 {-0.06/10 1s} Ne7 {+0.23/11 0s} 42. Bb4 {-0.42/11 1s} Qb3 {+0.63/10 0s}
43. Bxe7 {-0.56/12 1s} Qxe3+ {+0.55/10 0s} 44. Kh1 {-0.72/13 1s}
Rxe7 {+0.75/11 0s} 45. Rxf5 {-0.68/12 1s} Rfe8 {+0.83/10 0s}
46. Rd5 {-0.85/11 1s} Qb3 {+1.07/11 1s} 47. Rd7 {-0.91/10 0s} Qxa4 {+1.00/11 0s}
48. Rxe7 {-0.88/11 1s} Rxe7 {+1.00/6 0s} 49. Qc8+ {-0.98/10 1s}
Kh7 {+1.05/12 0s} 50. Qf5+ {-1.22/11 1s} g6 {+1.20/13 1s} 51. Qd5 {-1.24/12 0s}
Kg7 {+1.22/12 0s} 52. Qxb7 {-1.06/10 1s} Qxd4 {+1.32/11 1s}
53. Qa8 {-1.18/10 1s} Qd7 {+1.30/11 0s} 54. Rf8 {-0.61/7 0s} Qe6 {+1.51/9 0s}
55. Rf4 {-1.13/8 1s} e3 {+2.14/10 0s} 56. Rf1 {-2.79/11 2s} e2 {+2.42/10 0s}
57. Re1 {-3.58/12 1s} Qf6 {+2.42/12 0s} 58. Qg2 {-2.66/11 0s} Qxb2 {+2.82/11 0s}
59. Qf2 {-2.77/10 0s} Qe5 {+3.00/11 1s} 60. Kg2 {-3.06/11 0s} c5 {+3.01/11 0s}
61. Qf4 {-2.93/12 0s} Qd4 {+3.01/11 0s} 62. h4 {-3.06/10 1s} Qc3 {+2.93/11 1s}
63. Qf2 {-3.06/9 0s} Qe5 {+2.86/13 1s} 64. Qf4 {-3.40/11 0s} Qd5+ {+2.86/12 1s}
65. Kg1 {-3.33/9 0s} c4 {+3.52/10 1s} 66. Kh2 {-3.74/6 0s} Qc5 {+3.91/11 0s}
67. Qc1 {-4.53/7 0s} c3 {+4.06/11 1s} 68. Kg2 {-4.81/9 0s} Re3 {+5.28/10 1s}
69. Kf2 {-4.77/8 0s} Rd3+ {+10.45/11 1s} 70. Kg2 {-7.36/10 0s}
Qd5+ {+11.94/11 1s} 71. Kh2 {-10.62/12 1s} Qe5 {+11.07/13 0s}
72. Rg1 {-8.84/6 0s} Rxg3 {+11.98/15 1s} 73. Rxg3 {-7.42/1 0s}
e1=Q {+11.98/15 0s} 74. Qxe1 {-7.42/1 0s} Qxe1 {+11.98/16 1s}
75. Rg4 {-8.14/1 0s} c2 {+11.98/12 0s} 76. Rg1 {-11.87/1 0s}
Qxh4+ {+22.17/12 0s} {White loses on time} 0-1

Hi Werner,

In very fast time control I think the better time control to use is the one with increments to avoid time losses and to take care of the lag in Windows that Ilari mentioned. For example: tc=18+0.2 will have an average one minute per game if average number of moves per game is 60. You could even use tc=40/20+0.2 but I'm not sure if other engines could handle it.

Also if you want to test specific number of CPUs make sure that the engines are setup to use the number of cores accordingly as sometimes engines uses the maximum number of threads it could find by default. For example you could add: "uci/Threads=1" for Glaurung or "uci/Max CPUs=1" for Rybka, etc..

[Werner]

Thanks for the tipps, Edsel!

I will try it with the new version. I did my tests already using the tool of Marc.

[Gian-Carlo Pascutto]

When I put my laptop in standby, and resume, one of the engines in the matches flags, which is the expected result. However, instead of recovering cutechess-cli locks up completely, not responding to Ctl-C *and* SIGTERM. Only way out is a SIGKILL.

[ilari]

Cutechess-cli version 0.1.4 has been released with a good number of changes:

- Millisecond-precision time controls are now properly written in PGN files

- It's possible to use Cute Chess' engine configurations. This means that one can use the conf=engine_name option instead of "cmd=command dir=directory proto=protocol" options. The engine configuration file should be in ~/.config/cutechess/cutechess.ini on Linux, ~/.config/cutechess.org/cutechess.ini on Mac, and on Windows Vista in /Users/username/AppData/Roaming/cutechess/cutechess.ini. On Windows XP it's probably something like Vista. It looks like this:

Code: Select all

[engines]
1\name=Engine1
1\command=engine1_cmd
1\working_directory=engine1_dir
1\protocol=0
2\name=Engine2
2\command=engine2_cmd
2\working_directory=engine2_dir
2\protocol=1

Protocol 0 is Xboard and 1 is UCI.
To display a list of configured engines, run "cutechess-cli --engines".

- KxR style castling moves are now accepted in Polyglot books

- The syntax for setting UCI options was changed from "uci/option" to "uci.option"

- UCI options with empty defaults are now accepted. Eg. Naum 3.1 sends the NalimovPath with an empty default, which was previously rejected.

- The UCI_Opponent option is now used if an engine supports it

- d7d8=q style promotions are now accepted, even though they violate the protocol. At least Thinker sends promotions in this format.

- Finally great news for Windows users: I've written a custom class for interprocess communication on Windows. It reads new data instantly (in less than 1 millisecond), so the communication lag is completely gone. This means that superfast games (eg. 1 second per whole game) should now work even without a time increment, as long as the engine can keep up with the pace.

[ilari]

When I put my laptop in standby, and resume, one of the engines in the matches flags, which is the expected result. However, instead of recovering cutechess-cli locks up completely, not responding to Ctl-C *and* SIGTERM. Only way out is a SIGKILL.

The signal handling is not optimal currently. Could you try to reproduce the problem with the -debug option, and then post the last 20 or something lines of Cutechess-cli's output? That would give me a good idea of the running state of the application.

[ilari]

Also if you want to test specific number of CPUs make sure that the engines are setup to use the number of cores accordingly as sometimes engines uses the maximum number of threads it could find by default. For example you could add: "uci/Threads=1" for Glaurung or "uci/Max CPUs=1" for Rybka, etc..

You can also use the "cpus=cpu_count" option, which now supports all the variants of the UCI option that I'm aware of. It also works with Xboard engines that support the "cores" command.

[Marc Lacrosse]

The engine configuration file should be in ~/.config/cutechess/cutechess.ini on Linux, ~/.config/cutechess.org/cutechess.ini on Mac, and on Windows Vista in /Users/username/AppData/Roaming/cutechess/cutechess.ini. On Windows XP it's probably something like Vista.

May I suggest a slight modification for Windows XP (and Vista - what is this thing?) users?
I would prefer that cutechess first checks its own directory for its ini file and defaults to writing it locally in the same place.
Then cutechess-cli would continue to be perfectly "portable" for those who like myself go from PC to PC with everything on a USB drive.

- Finally great news for Windows users: I've written a custom class for interprocess communication on Windows. It reads new data instantly (in less than 1 millisecond)

Wonderful !

Marc