Page 1 of 5
To Ilari : CLOP + cutechess-cli
Posted: Sun Oct 09, 2011 2:19 pm
by mcostalba
Hi Ilari,
would be possible to interface cutechess-cli directly with CLOP ?
Currently some glue script is needed, but actually the so called DummyScript.py just parses the calling arguments with the parameters to tune, calls the match manager (for instance cutechess-cli) and reports back the match result in a way compatible with CLOP.
I am wondering about CLOP calling directly cuechess-cli with the CLOP parameters. Among these parameters there is one called "processor id" and can be used as a pointer to a cutechess JSON conf file with all the necessary configurations. All the others are the paremeter-value pair to be passed to the engines by the means of a UCI option.
Code: Select all
Arguments are:
#1: processor id (symbolic name, typically a machine name to ssh to)
#2: seed (integer)
#3: parameter id of first parameter (symbolic name)
#4: value of first parameter (float)
#5: parameter id of second parameter (optional)
#6: value of second parameter (optional)
...
In the JSON conf file, apart from engine setup data, could be added a is_CLOP flag so that cutechess-cli would know to print the output at the end of the game in the CLOP format.
This would be very useful so that any engine author that wants to try CLOP doesn't have to reinvent the wheel writing up yet another glue script.
What do you think ?
Marco
P.S: Perhaps another possibility is to write a clop_to_cute.py script that does the above, is distributed with cutechess-cli and is written once and for all.
Re: To Ilari : CLOP + cutechess-cli
Posted: Sun Oct 09, 2011 4:16 pm
by ilari
It sounds good. It has been my plan for a long time to add some kind of automatic parameter-tuning to Cutechess-cli. Just to make sure that I understood correcly:
- Cutechess-cli should be the "script", so in the .clop file we'd have the line "Script path_to_cutechess-cli/cutechess-cli.sh".
- Cutechess-cli should parse the CLOP parameters and send the values to the first engine (-fcp). For UCI engines this could be done with "setoption" or with a similar command.
- When the game or games have been played Cutechess-cli should just print "W", "L" or "D".
Did I get that right? If I did, it sure sounds easy. I'd just have to add a new parameter (eg. "-clop") that puts Cutechess-cli into a mode where it handles the CLOP parameters and outputs the result in CLOP's format. Cutechess-cli's normal parameters could be included in the "Script X" line of the .clop file, so the line could be something like:
Code: Select all
Script "path_to_cutechess-cli/cutechess-cli.sh -fcp conf=TunableEngine -scp conf=ReferenceEngine -both tc=10+0.1 -games 2"
If that works then there would be no need to mess with any JSON file.
Re: To Ilari : CLOP + cutechess-cli
Posted: Sun Oct 09, 2011 6:04 pm
by mcostalba
The key point here is that the script property of the clop file is able to keep both the path and the calling arguments, if this works I guess you described the easiest solution.
Re: To Ilari : CLOP + cutechess-cli
Posted: Sun Oct 09, 2011 7:11 pm
by ilari
mcostalba wrote:The key point here is that the script property of the clop file is able to keep both the path and the calling arguments, if this works I guess you described the easiest solution.
I just tested it with the dummy experiment, and it works without the quotation marks around the Script command. So you could have this line in the .clop file:
Code: Select all
Script path_to_cutechess-cli/cutechess-cli.sh -fcp conf=TunableEngine -scp conf=ReferenceEngine -both tc=10+0.1 -clop
CLOP's parameters would then be appended after the regular parameters, and would be easy to parse in Cutechess-cli.
Re: To Ilari : CLOP + cutechess-cli
Posted: Sun Oct 09, 2011 7:14 pm
by Edmund
I would appreciate a CLOP Feature of cutechess-cli very much myself. But please don't forget about the seed.
The seed is used for two aspects.
1) First-mover advantage:
usually you want your testconfigurations play two games in a row, one for white and one for black, just to cancel out the first-mover advantage.
2) Gauntlet:
usually you want your testconfiguration play against different opponents in a row.
With CLOP these aspects are handles with a single seed variable that gets passed to the script.
I suggest cutechess to interpret this seed as:
- Switch sides if (seed & 1)
- Play the n'th opponent in the list, where n = seed >> 1
BTW, a more elaborate solution would be to integrate CLOP directly into the cutechess compile. IIRC the license would permit that. But the way described by Marco sounds perfectly fine to me too.
Re: To Ilari : CLOP + cutechess-cli
Posted: Sun Oct 09, 2011 8:10 pm
by Rémi Coulom
ilari wrote:- When the game or games have been played Cutechess-cli should just print "W", "L" or "D".
Beware that CLOP expects only one game per script invocation.
Rémi
Re: To Ilari : CLOP + cutechess-cli
Posted: Sun Oct 09, 2011 9:17 pm
by mcostalba
ilari wrote: I just tested it with the dummy experiment, and it works without the quotation marks around the Script command.
Great ! And what about running the testing engine in a gauntlet against different opponents ?
If the clop file supports specifying more than one script command that are called in a loop then we are done: each script for each reference engine, otherwise we need a dispacther somewhere perhaps a clop-to-cute.py between clop and cutechess-cli ?
If instead you want to generalize on cutechess you can define -fcp -scp -tcp -fcp and so on, or more cleanly the aliases -cp1 ... -cpN, cutechess will play a gauntlet (-glet) against the first engine or a round robbin (-rr), in our case a gauntlet is what we need.
Sorry to write all this, I am writing ideas as come to my mind, feel free to trash at your wishes
Re: To Ilari : CLOP + cutechess-cli
Posted: Sun Oct 09, 2011 9:21 pm
by Rémi Coulom
mcostalba wrote:ilari wrote: I just tested it with the dummy experiment, and it works without the quotation marks around the Script command.
Great ! And what about running the testing engine in a gauntlet against different opponents ?
If the clop file supports specifying more than one script command that are called in a loop then we are done: each script for each reference engine, otherwise we need a dispacther somewhere perhaps a clop-to-cute.py between clop and cutechess-cli ?
Edmund explained that. Clop can deal with gauntlet and color alternation with the "Replications" parameter. You can use "seed % Replications" to choose opponent or color.
Rémi
Re: To Ilari : CLOP + cutechess-cli
Posted: Sun Oct 09, 2011 9:31 pm
by mcostalba
Rémi Coulom wrote: Edmund explained that. Clop can deal with gauntlet and color alternation with the "Replications" parameter. You can use "seed % Replications" to choose opponent or color.
Rémi
The problem is that cutechess does not support a list of engines but just 2, so to make seed >> 1 work with cutechess you have somehow to tell cutechess about the engines list.
Re: To Ilari : CLOP + cutechess-cli
Posted: Sun Oct 09, 2011 10:23 pm
by ilari
Edmund wrote:I would appreciate a CLOP Feature of cutechess-cli very much myself. But please don't forget about the seed.
The seed is used for two aspects.
1) First-mover advantage:
usually you want your testconfigurations play two games in a row, one for white and one for black, just to cancel out the first-mover advantage.
2) Gauntlet:
usually you want your testconfiguration play against different opponents in a row.
With CLOP these aspects are handles with a single seed variable that gets passed to the script.
I suggest cutechess to interpret this seed as:
- Switch sides if (seed & 1)
- Play the n'th opponent in the list, where n = seed >> 1
At first I didn't exactly know what the seed was, until I actually tried CLOP. So yes, the seed will be used to choose the playing side and the opponent.
But I'm still pondering whether or not I should leave that stuff for a script that operates between CLOP and Cutechess-cli.
BTW, a more elaborate solution would be to integrate CLOP directly into the cutechess compile. IIRC the license would permit that. But the way described by Marco sounds perfectly fine to me too.
That would be quite tricky in many ways.