To Ilari : CLOP + cutechess-cli

[Michel]

BTW 692 samples is _much_ too low. You need at least 100000.

[Daniel Shawul]

I think I observed something similar. But I am not sure.

I think that is the case. For my first experiment I did overnight with 15000 games the parameters I found (270,400) are far from the default values (150,300). But through out the simulation , I didn't notice the mean value being anywhere near 150.

Well it can, but currently this requires some manual input. First generate samples in a small window around the suggested values. Then enlarge the window to a wide one.

Now that is a good idea. Using a (x - delta, x + delta) should find optimal value around user suggested value much faster. A black box optimizer should either do that or have an algorithm which scans the domain for possible optimal values before focusing effort on one. I implemented in the past a particle swarm algorithm + BFGS as a general black box optimizer.

Obviously sample size depends on the window size and also to how many elos precison you want to optimize. Usually I do mine with 3000 games that give me results within +/- 10 so 100000 is out of the question.

Code: Select all

Samples = 2171
TotalWeight = 1385.38
Wins = 639
Draws = 555
Losses = 977
WinningRate = 0.422156

Rerunning test with your suggestion of narrow windows +/-6 around guessed values seems to give much better results faster.
Edit:

Code: Select all

Samples = 767
TotalWeight = 497.017
Wins = 272
Draws = 247
Losses = 248
WinningRate = 0.515645

For almost same number of games it started winning. New paramters found for mobility 15, 17, 15, 17. So applying bias around suggested values saves a lot of wasted effort IMO.

[LucenaTheLucid]

Ok I am having some trouble getting Clop running through cutechess-cli. I have clop under C:\Clop\Clop build and cutechess-cli under C:\Clop\cutechess-cli. Of course all engines are under the engine folder C:\Clop\cutechess-cli\Engines. The Clop GUI starts up and outputs a log that looks like this:

Code: Select all

2011-10-16 19:59:45 ===== Starting: Experiment =====
2011-10-16 20:00:01 Run
2011-10-16 20:00:01 id = 0 (machine1); Seed = 0; Starting new sample
2011-10-16 20:00:02 id = 0 (machine1); Seed = 0; Done (2)
2011-10-16 20:00:02 id = 0 (machine1); Seed = 1; Starting new sample
2011-10-16 20:00:02 id = 0 (machine1); Seed = 1; Done (2)
2011-10-16 20:00:02 id = 0 (machine1); Seed = 2; Starting new sample
2011-10-16 20:00:02 id = 0 (machine1); Seed = 2; Done (2)
2011-10-16 20:00:02 id = 0 (machine1); Seed = 3; Starting new sample
2011-10-16 20:00:02 id = 0 (machine1); Seed = 3; Done (2)
2011-10-16 20:00:02 id = 0 (machine1); Seed = 4; Starting new sample
2011-10-16 20:00:02 id = 0 (machine1); Seed = 4; Done (2)
2011-10-16 20:00:02 id = 0 (machine1); Seed = 5; Starting new sample
2011-10-16 20:00:03 id = 0 (machine1); Seed = 5; Done (2)
2011-10-16 20:00:03 id = 0 (machine1); Seed = 6; Starting new sample
2011-10-16 20:00:03 id = 0 (machine1); Seed = 6; Done (2)

My only problem is that all the games are ending in "draw".
This is what my "Engines.json" file looks like:

Code: Select all

[
   {
      "command":"Glass_1.7.exe",
      "name":"Glass 1.7",
      "protocol":"uci",
      "workingDirectory":"C:\Clop\cutechess-cli\Engines"
   },

   {
       "command":"amyan.exe",
       "name":"Amyan 1.72",
       "protocol":"uci",
       "workingDirectory":"C:\Clop\cutechess-cli\Engines"
   },

   {
       "command":"Philou.exe",
       "name":"Philou 3.60",
       "protocol":"uci",
       "workingDirectory":"C:\Clop\cutechess-cli\Engines"
   },

   {
       "command":"Pupsi2.exe",
       "name":"Pupsi2 0.08",
       "protocol":"uci",
       "workingDirectory":"C:\Clop\cutechess-cli\Engines"
   },

   {
       "command":"rotor.exe",
       "name":"Rotor 0.6",
       "protocol":"uci",
       "workingDirectory":"C:\Clop\cutechess-cli\Engines"
   },

   {
       "command":"Ruffian_105.exe",
       "name":"Ruffian 1.05",
       "protocol":"uci",
       "workingDirectory":"C:\Clop\cutechess-cli\Engines"
   }
]

This is my "Experiment.clop". Of course my computer is only 1 cpu.

Code: Select all

#
# Experiment.clop
#

# Name (used for .log and .dat files)
Name Experiment

# Script for running a game. See DummyScript.py for details.
Script "C:/Python27/python.exe" C:/Clop/cutechess-cli/x.py

# Parameter(s) to be optimized
# <parameter_type> <name> <min> <max>
# <parameter_type> may be&#58;
#  LinearParameter
#  IntegerParameter
#  GammaParameter
#  IntegerGammaParameter
# For GammaParameter, quadratic regression is performed on log&#40;x&#41;
# Warning&#58; 123 and not 123.0 should be used for IntegerParameter
IntegerParameter eLMR_mind 		0	10
IntegerParameter eLMR_depth 		0 	10
IntegerParameter eLMR_extend_fact 	0 	20
IntegerParameter eLMR_extend_static	0 	20

# This could be the list of machine names in a distributed experiment.
# In order to run 4 games in parallel, 2 on machine1, 2 on machine2&#58; 
Processor machine1

# Call the script "Replications" times in a row with identical parameter values
# Replications may be used to alternate colors, for instance.
# Seed % Replications would indicate color.
Replications 1

# Parameters of statistical model of outcome
# For binary outcome &#40;Win/Loss, no draws&#41;, use "DrawElo 0"
# For chess, use "DrawElo 100"
DrawElo 100

# Regression parameters
# H 3 is recommended &#40;it is the default value&#41;
# Correlations may be "all" &#40;default&#41; or "none"
# Even if variables are not correlated "all" should work well. The problem is
# that the regression might become very costly if the number of variables is
# high. So use "Correlations none" only if you are certain parameters are
# independent or you have so many variables that "all" is too costly.
H 3
Correlations all

And this is my "x.py" file:

Code: Select all

#!/usr/bin/env python
"""
 clop-cutechess-cli.py

 This script works between CLOP and cutechess-cli. Path to this script should
 be on the "Script" line of the .clop file. The settings 'cutechess_cli_path',
 'engine', 'engine_param_cmd', 'opponents' and 'options' should be modified
 according to the test environment and conditions.

 Arguments are:
  #1: processor id (symbolic name, typically a machine name to ssh to)
  #2: seed (integer)
  #3: parameter id of first parameter (symbolic name)
  #4: value of first parameter (float)
  #5: parameter id of second parameter (optional)
  #6: value of second parameter (optional)
  ...

 This script writes the game outcome to its standard output:
  W = win
  L = loss
  D = draw
"""

import os
from subprocess import Popen, PIPE
import subprocess 
import sys
import exceptions




# Path to the cutechess-cli executable.
# On Windows this should point to cutechess-cli.exe
cutechess_cli_path = 'C:/Clop/cutechess-cli/cutechess-cli.exe'

# The engine whose parameters will be optimized
engine = 'conf="Glass 1.7"'

# Format for the commands that are sent to the engine to
# set the parameter values. When the command is sent,
# {name} will be replaced with the parameter name and {value}
# with the parameter value.
engine_param_cmd = 'setoption name {name} value {value}'
#engine_param_cmd = 'option.{name}="{value}"'

# A pool of opponents for the engine. The opponent will be
# chosen based on the seed sent by CLOP.
opponents = [
    'conf="Amyan 1.72"',
    'conf="Rotor 0.6"',
    'conf="Pupsi2 0.08"',
    'conf="Ruffian 1.05"',
    'conf="Philou 3.60"'
]

# Additional cutechess-cli options, eg. time control and opening book
options = '-both tc=120/1+0.1 -draw 80 1 -resign 10 500 -recover -pgnout Glass99.pgn'


def main(argv = None):
    if argv is None:
        argv = sys.argv[2:]

    if len&#40;argv&#41; < 3 or len&#40;argv&#41; % 2 == 0&#58;
        print 'Too few arguments'
        return 2
    
    clop_seed = int&#40;argv&#91;0&#93;)
    fcp = engine
    scp = opponents&#91;&#40;clop_seed >> 1&#41; % len&#40;opponents&#41;&#93;
    
    # Parse the parameters that should be optimized
    for i in range&#40;1, len&#40;argv&#41;, 2&#41;&#58;
        # Make sure the parameter value is numeric
        try&#58;
            tmp = float&#40;argv&#91;i + 1&#93;)
        except exceptions.ValueError&#58;
            print 'Invalid value for parameter ' + argv&#91;i&#93; + '&#58; ' + argv&#91;i + 1&#93;
            return 2
        # Pass CLOP's parameters to the engine by using
        # cutechess-cli's initialization string feature
        initstr = engine_param_cmd.format&#40;name = argv&#91;i&#93;, value = argv&#91;i + 1&#93;)
        fcp += ' initstr="' + initstr + '"'
        #fcp += ' ' + initstr
    
    # Choose the engine's playing side &#40;color&#41; based on CLOP's seed
    if clop_seed % 2 != 0&#58;
        fcp, scp = scp, fcp
    
    cutechess_args = '-fcp ' + fcp + ' -scp ' + scp + ' ' + options
    command = cutechess_cli_path + ' ' + cutechess_args
	
    #print command
    
    # Run cutechess-cli and wait for it to finish
    #process = Popen&#40;command, shell = True, stdout = PIPE&#41;
    process = subprocess.Popen&#40;command, shell = True, stdout = PIPE&#41; 
    process.wait&#40;) 
    #sts = os.waitpid&#40;process.pid, 0&#41;
    #if sts&#91;1&#93; != 0&#58;
    #    print 'Could not execute command&#58; ' + command
    #    return 2
    
    # Convert Cutechess-cli's result into W/L/D
    # Note that only one game should be played
    result = -1
    for line in process.stdout&#58;
        if line.startswith&#40;'Game 1 ended&#58;')&#58;
            if line.find&#40;"1-0") != -1&#58;
                result = clop_seed % 2
            elif line.find&#40;"0-1") != -1&#58;
                result = &#40;clop_seed % 2&#41; ^ 1
            break
    
    if result == -1&#58;
        print 'D'
    elif result == 0&#58;
        print 'W'
    elif result == 1&#58;
        print 'L'
		
if __name__ == "__main__"&#58;
    sys.exit&#40;main&#40;))

[Daniel Shawul]

It could be because you don't have a book

Code: Select all

# Additional cutechess-cli options, eg. time control and opening book 
options = '-both tc=120/1+0.1 -draw 80 1 -resign 10 500 -recover -pgnout Glass99.pgn' 

[ilari]

Ok I am having some trouble getting Clop running through cutechess-cli. I have clop under C:\Clop\Clop build and cutechess-cli under C:\Clop\cutechess-cli. Of course all engines are under the engine folder C:\Clop\cutechess-cli\Engines.

A few notes:

• The engine config file must be named "engines.json" with all lowercase, not "Engines.json"
• Backslashes need to be escaped with another backslash in engines.json. So the "workingDirectory" line needs to look like this:

  Code: Select all

  "workingDirectory" : "C:\\Clop\\cutechess-cli\\Engines"

• You should set "Replications" to 2 in the .clop file.
• Please download the "clop-cutechess-cli.py" script again and use the new one. Your current script doesn't care if cutechess-cli exited normally or with an error. You can replace anything above the "def main(argv = None)" line with what you have now.

[Ferdy]

Okay, I decided to go with the script approach. It was just the better and easier option in many ways. Here: http://koti.mbnet.fi/~ilaripih/bin/clop ... ess-cli.py

It's in Python, so those running Windows may have to install the Python implementation from www.python.org first.

The script has to be modified a bit before it can be used. The relevant parts are at the beginning of the script:

Code: Select all

# Path to the cutechess-cli executable.
# On Windows this should point to cutechess-cli.exe
cutechess_cli_path = '$HOME/chess/cutechess-cli/cutechess-cli.sh'

# The engine whose parameters will be optimized
engine = 'conf=Sloppy-0.2.2'

# Format for the commands that are sent to the engine to
# set the parameter values. When the command is sent,
# {name} will be replaced with the parameter name and {value}
# with the parameter value.
engine_param_cmd = 'setvalue {name} {value}'

# A pool of opponents for the engine. The opponent will be
# chosen based on the seed sent by CLOP.
opponents = [
    'conf=Fruit-2.1',
    'conf=Crafty-22.10',
    'conf=Glaurung-2.2',
    'conf=Gaviota-0.8.0'
]

# Additional cutechess-cli options, eg. time control and opening book
options = '-both tc=40/1+0.05 book=$HOME/chess/pg_books/performance.bin -draw 80 1 -resign 5 500'

* <cutechess_cli_path> should be the full path to the cutechess-cli executable or launcher script.
* <engine> can be anything that you would normally put in '-fcp', so you can also use 'cmd=engine.exe dir=path_to_engine proto=xboard', etc.
* <engine_param_cmd> is the command that is sent to your engine, and you can change the syntax freely, as long as the {name} and {value} fields are there. To optimize UCI options you can set engine_option_cmd to 'setoption name {name} value {value}'. The option
* <opponents> is a list of opponents to play against. There has to be at least one opponent.
* <options> should have all the rest of cutechess-cli's parameters.

This script can be used with any Cutechess-cli version, so go ahead and give it a try.

Code: Select all

options = '-both tc=40/1+0.05 book=$HOME/chess/pg_books/performance.bin -draw 80 1 -resign 5 500' 

Thanks for your sample script, I am testing it now after successfully compiling clop for windows, BTW, how can I assign TC for engine to be tuned and a different TC for list of opponents? or different TC's for each opponents?

[ilari]

Thanks for your sample script, I am testing it now after successfully compiling clop for windows, BTW, how can I assign TC for engine to be tuned and a different TC for list of opponents? or different TC's for each opponents?

It's quite simple: just remove "tc" from "options" and instead append it to "engine" and "opponents" configurations. You would end up with something like this:

Code: Select all

engine = 'conf=Sloppy-0.2.2 tc=1+0.1'
opponents = [
    'conf=Fruit-2.1 tc=40/2',
    'conf=Crafty-22.10 tc=1+0.01',
    'conf=Glaurung-2.2 tc=2',
    'conf=Gaviota-0.8.0 tc=2'
]

options = '-both book=$HOME/chess/pg_books/performance.bin -draw 80 1 -resign 5 500' 

[Ferdy]

Thanks for your sample script, I am testing it now after successfully compiling clop for windows, BTW, how can I assign TC for engine to be tuned and a different TC for list of opponents? or different TC's for each opponents?

It's quite simple: just remove "tc" from "options" and instead append it to "engine" and "opponents" configurations. You would end up with something like this:

Code: Select all

engine = 'conf=Sloppy-0.2.2 tc=1+0.1'
opponents = [
    'conf=Fruit-2.1 tc=40/2',
    'conf=Crafty-22.10 tc=1+0.01',
    'conf=Glaurung-2.2 tc=2',
    'conf=Gaviota-0.8.0 tc=2'
]

options = '-both book=$HOME/chess/pg_books/performance.bin -draw 80 1 -resign 5 500' 

Nice thanks again

[mcostalba]

Okay, I decided to go with the script approach. It was just the better and easier option in many ways. Here: http://koti.mbnet.fi/~ilaripih/bin/clop ... ess-cli.py

Hi Ilari,

here I am again

Your script works flawless. Currently I run a game at a time, but because I have a QUAD I'd like to run 2 games in parallel, or even 4 in parallel in single thread no ponder, so to accumulate games faster.

I know CLOP foreseen the possibility to call clop-cutechess-cli.py many times without waiting for the game to finish:

Code: Select all

# This could be the list of machine names in a distributed experiment.
# In order to run 4 games in parallel, 2 on machine1, 2 on machine2: 
Processor machine1
Processor machine1
Processor machine2
Processor machine2

So my request is. Could be possible to modify the script so to support running more than one cutechess-cli sessions in parallel, according to the 'Processor' number ?

Thanks
Marco

[ilari]

Hi Ilari,

here I am again

Your script works flawless. Currently I run a game at a time, but because I have a QUAD I'd like to run 2 games in parallel, or even 4 in parallel in single thread no ponder, so to accumulate games faster.

I know CLOP foreseen the possibility to call clop-cutechess-cli.py many times without waiting for the game to finish:

Code: Select all

# This could be the list of machine names in a distributed experiment.
# In order to run 4 games in parallel, 2 on machine1, 2 on machine2: 
Processor machine1
Processor machine1
Processor machine2
Processor machine2

So my request is. Could be possible to modify the script so to support running more than one cutechess-cli sessions in parallel, according to the 'Processor' number ?

Thanks
Marco

Hi Marco,

CLOP already does take care of the concurrency stuff, at least it does on my system (64-bit Linux box with a dual-core processor). I just have this in my .clop file to run 2 games in parallel:

Code: Select all

Processor cpu1
Processor cpu2

Does that not work on your system, or did I misinterpret your post?