Cutechess Testing

[AAce3]

Hi all,

I am trying to configure Cutechess-cli testing for my engine. I downloaded the windows installer for cutechess, but I can't seem to find the CLI version with SPRT. In addition, I would like to configure it so that the starting position comes from a variety of openings. However, I have not implemented polyglot for my engine. Is there any workaround for this? Potentially starting from FENs or PGNs?

[JoAnnP38]

This is the .BAT file I used to run SPRT tests from windows. You will notice there is a file referenced in -opening (i.e. file=openings-10ply-100k.pgn) that is used to generate the opening sequences (randomly) for my SPRT matches. You can download this opening book plus many more using the link in step 3 of Lc0 testing. Also I encourage you to read through all the options documented in the file cutechess-cli.6.txt stored in your default cutechess installation folder (commonly this is C:\Program Files (x86)\Cute Chess in windows.)

Code: Select all

@ECHO OFF
SET OG_PATH=%PATH%
SET PATH=C:\Program Files (x86)\Cute Chess;%PATH%
SET CUR_YYYY=%date:~10,4%
SET CUR_MM=%date:~4,2%
SET CUR_DD=%date:~7,2%
SET PGNFILE=sprt-test-%CUR_YYYY%-%CUR_MM%-%CUR_DD%.PGN

cutechess-cli ^
 -engine conf=Pedantic_Dev name="Pedantic Dev" ^
 -engine conf=Pedantic_Base name="Pedantic Base" ^
 -each tc=20+0.2 restart=on timemargin=20 ^
 -openings file="openings-10ply-100k.pgn" format=pgn order=random plies=10 policy=default ^
 -games 25000 -repeat 2 -resultformat wide -recover -wait 20 ^
 -maxmoves 200 -tb c:/tb/syzygy/3-4-5 -tbpieces 5 ^
 -ratinginterval 10 -variant standard -concurrency 16 ^
 -sprt elo0=0 elo1=3 alpha=0.1 beta=0.1 ^
 -event sprt-test -pgnout "%PGNFILE%" -site "Clearwater, FL USA" -tournament round-robin
	
SET PATH=%OG_PATH%

[lithander]

However, I have not implemented polyglot for my engine. Is there any workaround for this? Potentially starting from FENs or PGNs?

Openings are typically handled by the GUI so your engine doesn't need the capability to read opening books.

E.g. in CuteChess GUI as part of the New Game wizzard you can provide an opening book in Polyglot format and the amount of moves and your engine will receive a start position like so:

Code: Select all

position startpos moves g2g3 d7d5 f1g2 c7c6 g1f3 c8g4 d2d3 b8d7 e1g1 e7e5 c2c4 d5c4

before asked the first time to provide a move using the 'go' command.

[Ajedrecista]

Hello Aaron:

[...] but I can't seem to find the CLI version with SPRT. [...]

Should not be in the Releases tab of the Github project?

Releases · cutechess/cutechess

I downloaded the current last version (1.3.1), more exactly the zip named cutechess-1.3.1-win64.zip, and there is a cutechess-cli.exe file once you unzip the file along with a readme file cutechess-cli.6.txt that says:

Code: Select all

[...]

     -sprt elo0=E0 elo1=E1 alpha=<alpha> beta=<beta>
	     Use a Sequential Probability Ratio Test as a termination
	     criterion for the match.

	     This option should only be used in matches between two players to
	     test if engine P1 is stronger than engine P2.  Hypothesis H1 is
	     that P1 is stronger than P2 by at least E0 ELO points, and H0
	     (the null hypothesis) is that P1 is not stronger than P2 by at
	     least E1 ELO points.  The maximum probabilities for type I and
	     type II errors outside the interval [ E0, E1 ] are <alpha> and
	     <beta>.

	     The match is stopped if either H0 or H1 is accepted or if the
	     maximum number of games set by -rounds and / or -games is
	     reached.
 
[...]

Link:

cutechess-1.3.1-win64.zip (33.8 MB)

Regards from Spain.

Ajedrecista.

[AAce3]

Hi all, thanks for the responses. I managed to get it working - it was mostly about debugging my understanding of windows CLIs .