CLOP in cutechess-cli debug

Discussion of chess software programming and technical issues.

Moderators: hgm, Rebel, chrisw

Albert Silver
Posts: 3019
Joined: Wed Mar 08, 2006 9:57 pm
Location: Rio de Janeiro, Brazil

CLOP in cutechess-cli debug

Post by Albert Silver »

I'm trying to get CLOP working but can't seem to get it started but no idea why. I get this error message:

2018-05-21 23:32:24 ===== Starting: Testing =====
2018-05-21 23:32:29 Run
2018-05-21 23:32:29 id = 0 (machine1); Seed = 0; Starting new sample
2018-05-21 23:32:29 id = 0 (machine1); Seed = 0; Error: No Outcome.
StandardOutput:

StandardError:
Warning: Unknown option: "machine1"

2018-05-21 23:32:29 id = 0 stopped. RunningProcessors = 0
2018-05-21 23:32:29 Stop

I'm guessing I am doing something silly, but cannot figure out what. Can anyone help? Here is my CLOP file:

Code: Select all

# See DummExperiment.clop for documentation about file format
Script C:/Users/Albert/Chess/cutechess/cutechess-cli.exe
Name Testing
LinearParameter Cpuct_MCTS_option 1.2 4.0
LinearParameter First_Play_Urgency_Reduction -1.0 1.0
Processor machine1
Replications 2
DrawElo 100
H 3
Correlations all
And this is the .py file:

Code: Select all

#!/usr/bin/python
# -*- coding: utf-8 -*-

from subprocess import Popen, PIPE
import sys

# Path to the cutechess-cli executable.
# On Windows this should point to cutechess-cli.exe
cutechess_cli_path = "C:/Users/Albert/Chess/cutechess/cutechess-cli.exe"

# The engine whose parameters will be optimized
engine = 'cmd=C:/Users/Albert/Chess/LCZero/lc0-may21.exe name=lc0 option."Network weights file path"=C:/Users/Albert/Chess/LCZero/weights314.txt option."Number of worker threads"=4'

# Format for the commands that are sent to the engine to
# set the parameter values. When the command is sent,
# {name} will be replaced with the parameter name and {value}
# with the parameter value.
engine_param_cmd = 'setoption name {name} value {value}'

# A pool of opponents for the engine. The opponent will be
# chosen based on the seed sent by CLOP.
opponents = [
    'cmd=C:/Users/Albert/Chess/wasp_300/wasp300-x64.exe name=Wasp3 option."Threads"=4'
]

# Additional cutechess-cli options, eg. time control and opening book
options = '-resign movecount=2 score=700 -draw movenumber=0 movecount=100 score=10 -openings file=C:/Users/Albert/Chess/ss-large-990.pgn -srand 42 -each proto=uci restart=off tc=1+1'


def main(argv = None):
    print('args:', argv, flush=True)
    if argv is None:
        argv = sys.argv[1:]

    if len(argv) == 0 or argv[0] == '--help':
        sys.stdout.write(__doc__)
        return 0

    argv = argv[1:]
    if len(argv) < 3 or len(argv) % 2 == 0:
        sys.stderr.write('Too few arguments\n')
        return 2

    clop_seed = 0
    try:
        clop_seed = int(argv[0])
    except ValueError:
        sys.stderr.write('invalid seed value: %s\n' % argv[0])
        return 2

    fcp = engine
    scp = opponents[(clop_seed >> 1) % len(opponents)]

    # Parse the parameters that should be optimized
    for i in range(1, len(argv), 2):
        # Make sure the parameter value is numeric
        try:
            float(argv[i + 1])
        except ValueError:
            sys.stderr.write('invalid value for parameter %s: %s\n' % (argv[i], argv[i + 1]))
            return 2
        # Pass CLOP's parameters to the engine by using
        # cutechess-cli's initialization string feature
        initstr = engine_param_cmd.format(name = argv[i].replace('_', ' '), value = argv[i + 1])
        fcp += ' initstr="%s"' % initstr

    # Choose the engine's playing side (color) based on CLOP's seed
    if clop_seed % 2 != 0:
        fcp, scp = scp, fcp

    cutechess_args = '-engine %s -engine %s %s' % (fcp, scp, options)
    command = '%s %s' % (cutechess_cli_path, cutechess_args)

    # Run cutechess-cli and wait for it to finish
    process = Popen(command, shell = True, stdout = PIPE)
    output = process.communicate()[0]
    if process.returncode != 0:
        sys.stderr.write('failed to execute command: %s\n' % command)
        return 2

    # Convert Cutechess-cli's result into W/L/D
    # Note that only one game should be played
    result = -1
    for line in output.decode("utf-8").splitlines():
        if line.startswith('Finished game'):
            if line.find(": 1-0") != -1:
                result = clop_seed % 2
            elif line.find(": 0-1") != -1:
                result = (clop_seed % 2) ^ 1
            elif line.find(": 1/2-1/2") != -1:
                result = 2
            else:
                sys.stderr.write('the game did not terminate properly\n')
                return 2
            break

    if result == 0:
        sys.stdout.write('W\n')
    elif result == 1:
        sys.stdout.write('L\n')
    elif result == 2:
        sys.stdout.write('D\n')

if __name__ == "__main__":
    sys.exit(main())
"Tactics are the bricks and sticks that make up a game, but positional play is the architectural blueprint."
Ferdy
Posts: 4833
Joined: Sun Aug 10, 2008 3:15 pm
Location: Philippines

Re: CLOP in cutechess-cli debug

Post by Ferdy »

On your clop file you should have something like the following.

Code: Select all

Script "C:/Python27/python.exe" C:/chess/clop-cutechess-cli.py
Albert Silver
Posts: 3019
Joined: Wed Mar 08, 2006 9:57 pm
Location: Rio de Janeiro, Brazil

Re: CLOP in cutechess-cli debug

Post by Albert Silver »

Ferdy wrote: Tue May 22, 2018 5:38 am On your clop file you should have something like the following.

Code: Select all

Script "C:/Python27/python.exe" C:/chess/clop-cutechess-cli.py
That did the trick, so many thanks. I am having trouble with the UCI options though when configuring the engines. Some are more than one word and have spaces. Any idea how best to solve this?
"Tactics are the bricks and sticks that make up a game, but positional play is the architectural blueprint."
AlvaroBegue
Posts: 931
Joined: Tue Mar 09, 2010 3:46 pm
Location: New York
Full name: Álvaro Begué (RuyDos)

Re: CLOP in cutechess-cli debug

Post by AlvaroBegue »

Albert Silver wrote: Tue May 22, 2018 6:00 pm I am having trouble with the UCI options though when configuring the engines. Some are more than one word and have spaces. Any idea how best to solve this?
Why is that a problem? The UCI protocol allows for multiple words in the name of options.
Ferdy
Posts: 4833
Joined: Sun Aug 10, 2008 3:15 pm
Location: Philippines

Re: CLOP in cutechess-cli debug

Post by Ferdy »

Albert Silver wrote: Tue May 22, 2018 6:00 pm
Ferdy wrote: Tue May 22, 2018 5:38 am On your clop file you should have something like the following.

Code: Select all

Script "C:/Python27/python.exe" C:/chess/clop-cutechess-cli.py
That did the trick, so many thanks. I am having trouble with the UCI options though when configuring the engines. Some are more than one word and have spaces. Any idea how best to solve this?
I usually use engines.json file.

Create engines.json file located on cutechess-cli.exe
Add entries like the following.

Code: Select all

[
    { 
      "command" : "stockfish_9_x64_popcnt.exe",
      "name" : "Stockfish_9",
      "options" : [ 
         { 
            "name" : "Hash",
            "value" : 128
         }		 
      ], 
      "protocol" : "uci",
      "workingDirectory" : "C:/chess/engines/nobook/stockfish-9-win/Windows"
    },
    { 
      "command" : "Hiarcs14WCSC.exe", 
      "name" : "HIARCS_14_UCI_Elo_1600", 
      "options" : [ 
         { 
            "name" : "Hash", 
            "value" : 64 
         },
	 { 
            "name" : "Book Learning", 
            "value" : false 
         },
	 { 
            "name" : "Use Tablebases", 
            "value" : "Never"
         },
         { 
            "name" : "UCI_LimitStrength", 
            "value" : true
         },
         { 
            "name" : "UCI_Elo", 
            "value" : 1600
         },
         { 
            "name" : "OwnBook", 
            "value" : false
         }
      ], 
      "protocol" : "uci", 
      "workingDirectory" : "C:/Program Files (x86)/HIARCS Chess/HIARCS 14 WCSC" 
   }
]
Then on python script just use,

Code: Select all

# The engine whose parameters will be optimized
engine = 'conf=HIARCS_14_UCI_Elo_1600'

# A pool of opponents for the engine. The opponent will be
# chosen based on the seed sent by CLOP.
opponents = [
    'conf=Stockfish_9'
]
Albert Silver
Posts: 3019
Joined: Wed Mar 08, 2006 9:57 pm
Location: Rio de Janeiro, Brazil

Re: CLOP in cutechess-cli debug

Post by Albert Silver »

Ferdy wrote: Tue May 22, 2018 8:58 pm
Albert Silver wrote: Tue May 22, 2018 6:00 pm
Ferdy wrote: Tue May 22, 2018 5:38 am On your clop file you should have something like the following.

Code: Select all

Script "C:/Python27/python.exe" C:/chess/clop-cutechess-cli.py
That did the trick, so many thanks. I am having trouble with the UCI options though when configuring the engines. Some are more than one word and have spaces. Any idea how best to solve this?
I usually use engines.json file.

Create engines.json file located on cutechess-cli.exe
Add entries like the following.

Code: Select all

[
    { 
      "command" : "stockfish_9_x64_popcnt.exe",
      "name" : "Stockfish_9",
      "options" : [ 
         { 
            "name" : "Hash",
            "value" : 128
         }		 
      ], 
      "protocol" : "uci",
      "workingDirectory" : "C:/chess/engines/nobook/stockfish-9-win/Windows"
    },
    { 
      "command" : "Hiarcs14WCSC.exe", 
      "name" : "HIARCS_14_UCI_Elo_1600", 
      "options" : [ 
         { 
            "name" : "Hash", 
            "value" : 64 
         },
	 { 
            "name" : "Book Learning", 
            "value" : false 
         },
	 { 
            "name" : "Use Tablebases", 
            "value" : "Never"
         },
         { 
            "name" : "UCI_LimitStrength", 
            "value" : true
         },
         { 
            "name" : "UCI_Elo", 
            "value" : 1600
         },
         { 
            "name" : "OwnBook", 
            "value" : false
         }
      ], 
      "protocol" : "uci", 
      "workingDirectory" : "C:/Program Files (x86)/HIARCS Chess/HIARCS 14 WCSC" 
   }
]
Then on python script just use,

Code: Select all

# The engine whose parameters will be optimized
engine = 'conf=HIARCS_14_UCI_Elo_1600'

# A pool of opponents for the engine. The opponent will be
# chosen based on the seed sent by CLOP.
opponents = [
    'conf=Stockfish_9'
]
Many thanks again, that was super helpful. I think I need to restart my CLOP session since I suspect I completely misunderstood the TC setting and thought tc=1+1 means 1m+1s and now think it is all seconds... It would explain why the games are going to fast.
"Tactics are the bricks and sticks that make up a game, but positional play is the architectural blueprint."
Albert Silver
Posts: 3019
Joined: Wed Mar 08, 2006 9:57 pm
Location: Rio de Janeiro, Brazil

Re: CLOP in cutechess-cli debug

Post by Albert Silver »

Ferdy wrote: Tue May 22, 2018 8:58 pm
Albert Silver wrote: Tue May 22, 2018 6:00 pm
Ferdy wrote: Tue May 22, 2018 5:38 am On your clop file you should have something like the following.

Code: Select all

Script "C:/Python27/python.exe" C:/chess/clop-cutechess-cli.py
That did the trick, so many thanks. I am having trouble with the UCI options though when configuring the engines. Some are more than one word and have spaces. Any idea how best to solve this?
I usually use engines.json file.

Create engines.json file located on cutechess-cli.exe
Add entries like the following.

Code: Select all

[
    { 
      "command" : "stockfish_9_x64_popcnt.exe",
      "name" : "Stockfish_9",
      "options" : [ 
         { 
            "name" : "Hash",
            "value" : 128
         }		 
      ], 
      "protocol" : "uci",
      "workingDirectory" : "C:/chess/engines/nobook/stockfish-9-win/Windows"
    },
    { 
      "command" : "Hiarcs14WCSC.exe", 
      "name" : "HIARCS_14_UCI_Elo_1600", 
      "options" : [ 
         { 
            "name" : "Hash", 
            "value" : 64 
         },
	 { 
            "name" : "Book Learning", 
            "value" : false 
         },
	 { 
            "name" : "Use Tablebases", 
            "value" : "Never"
         },
         { 
            "name" : "UCI_LimitStrength", 
            "value" : true
         },
         { 
            "name" : "UCI_Elo", 
            "value" : 1600
         },
         { 
            "name" : "OwnBook", 
            "value" : false
         }
      ], 
      "protocol" : "uci", 
      "workingDirectory" : "C:/Program Files (x86)/HIARCS Chess/HIARCS 14 WCSC" 
   }
]
Then on python script just use,

Code: Select all

# The engine whose parameters will be optimized
engine = 'conf=HIARCS_14_UCI_Elo_1600'

# A pool of opponents for the engine. The opponent will be
# chosen based on the seed sent by CLOP.
opponents = [
    'conf=Stockfish_9'
]
I'm really mystified now. I followed your instructions, and even successfully ran the command line:

C:\Users\Albert\Chess\cutechess>cutechess-cli -engine conf=Wasp -engine conf=lc0-may22 -each proto=uci tc=60+1
Started game 1 of 1 (Wasp vs lc0-may22)
Finished game 1 (Wasp vs lc0-may22): 0-1 {Black mates}
Score of Wasp vs lc0-may22: 0 - 1 - 0 [0.000] 1
Elo difference: -inf +/- nan
Finished match


I did this to make certain the engines.json file was correct. Then I added the lines to the py as suggested:

Code: Select all

# The engine whose parameters will be optimized
engine = 'conf=lc0-may22'

# Format for the commands that are sent to the engine to
# set the parameter values. When the command is sent,
# {name} will be replaced with the parameter name and {value}
# with the parameter value.
engine_param_cmd = 'setoption name {name} value {value}'

# A pool of opponents for the engine. The opponent will be
# chosen based on the seed sent by CLOP.
opponents = [
    'conf=Wasp'
]

# Additional cutechess-cli options, eg. time control and opening book
options = '-resign movecount=5 score=700 -draw movenumber=80 movecount=100 score=10 -openings file=C:/Users/Albert/Chess/ss-large-990.pgn -srand 42 -each proto=uci restart=off tc=60+1'
But cannot run as it crashed with the error message:

2018-05-23 20:55:26 ===== Starting: FPU-PUCT-Testing =====
2018-05-23 20:55:30 Run
2018-05-23 20:55:30 id = 0 (machine1); Seed = 0; Starting new sample
2018-05-23 20:55:30 id = 0 (machine1); Seed = 0; Error: No Outcome.
StandardOutput:

StandardError:
Warning: Unknown engine configuration: "Wasp"

Warning: Invalid value for option "-engine": "conf=Wasp"


Any ideas?
"Tactics are the bricks and sticks that make up a game, but positional play is the architectural blueprint."
Milos
Posts: 4190
Joined: Wed Nov 25, 2009 1:47 am

Re: CLOP in cutechess-cli debug

Post by Milos »

Albert Silver wrote: Thu May 24, 2018 2:05 am I'm really mystified now. I followed your instructions, and even successfully ran the command line:

C:\Users\Albert\Chess\cutechess>cutechess-cli -engine conf=Wasp -engine conf=lc0-may22 -each proto=uci tc=60+1
Started game 1 of 1 (Wasp vs lc0-may22)
Finished game 1 (Wasp vs lc0-may22): 0-1 {Black mates}
Score of Wasp vs lc0-may22: 0 - 1 - 0 [0.000] 1
Elo difference: -inf +/- nan
Finished match


I did this to make certain the engines.json file was correct. Then I added the lines to the py as suggested:

Code: Select all

# The engine whose parameters will be optimized
engine = 'conf=lc0-may22'

# Format for the commands that are sent to the engine to
# set the parameter values. When the command is sent,
# {name} will be replaced with the parameter name and {value}
# with the parameter value.
engine_param_cmd = 'setoption name {name} value {value}'

# A pool of opponents for the engine. The opponent will be
# chosen based on the seed sent by CLOP.
opponents = [
    'conf=Wasp'
]

# Additional cutechess-cli options, eg. time control and opening book
options = '-resign movecount=5 score=700 -draw movenumber=80 movecount=100 score=10 -openings file=C:/Users/Albert/Chess/ss-large-990.pgn -srand 42 -each proto=uci restart=off tc=60+1'
But cannot run as it crashed with the error message:

2018-05-23 20:55:26 ===== Starting: FPU-PUCT-Testing =====
2018-05-23 20:55:30 Run
2018-05-23 20:55:30 id = 0 (machine1); Seed = 0; Starting new sample
2018-05-23 20:55:30 id = 0 (machine1); Seed = 0; Error: No Outcome.
StandardOutput:

StandardError:
Warning: Unknown engine configuration: "Wasp"

Warning: Invalid value for option "-engine": "conf=Wasp"


Any ideas?
It's pretty simple, you defined only 1 engine in json file - LC0. You didn't define Wasp. You have to define both.
Just read the manual for cutechess-cli. You have 2 txt files in cutechess folder where everything is explained.
Albert Silver
Posts: 3019
Joined: Wed Mar 08, 2006 9:57 pm
Location: Rio de Janeiro, Brazil

Solved it - thanks

Post by Albert Silver »

Milos wrote: Thu May 24, 2018 2:10 am
Albert Silver wrote: Thu May 24, 2018 2:05 am I'm really mystified now. I followed your instructions, and even successfully ran the command line:

C:\Users\Albert\Chess\cutechess>cutechess-cli -engine conf=Wasp -engine conf=lc0-may22 -each proto=uci tc=60+1
Started game 1 of 1 (Wasp vs lc0-may22)
Finished game 1 (Wasp vs lc0-may22): 0-1 {Black mates}
Score of Wasp vs lc0-may22: 0 - 1 - 0 [0.000] 1
Elo difference: -inf +/- nan
Finished match


I did this to make certain the engines.json file was correct. Then I added the lines to the py as suggested:

Code: Select all

# The engine whose parameters will be optimized
engine = 'conf=lc0-may22'

# Format for the commands that are sent to the engine to
# set the parameter values. When the command is sent,
# {name} will be replaced with the parameter name and {value}
# with the parameter value.
engine_param_cmd = 'setoption name {name} value {value}'

# A pool of opponents for the engine. The opponent will be
# chosen based on the seed sent by CLOP.
opponents = [
    'conf=Wasp'
]

# Additional cutechess-cli options, eg. time control and opening book
options = '-resign movecount=5 score=700 -draw movenumber=80 movecount=100 score=10 -openings file=C:/Users/Albert/Chess/ss-large-990.pgn -srand 42 -each proto=uci restart=off tc=60+1'
But cannot run as it crashed with the error message:

2018-05-23 20:55:26 ===== Starting: FPU-PUCT-Testing =====
2018-05-23 20:55:30 Run
2018-05-23 20:55:30 id = 0 (machine1); Seed = 0; Starting new sample
2018-05-23 20:55:30 id = 0 (machine1); Seed = 0; Error: No Outcome.
StandardOutput:

StandardError:
Warning: Unknown engine configuration: "Wasp"

Warning: Invalid value for option "-engine": "conf=Wasp"


Any ideas?
It's pretty simple, you defined only 1 engine in json file - LC0. You didn't define Wasp. You have to define both.
Just read the manual for cutechess-cli. You have 2 txt files in cutechess folder where everything is explained.
Actually, the answer was other, but your explanation was plausible, so thanks. I had placed the engines.json file in the cutechess folder, but for CLOP, it needs to be in the CLOP folder.
"Tactics are the bricks and sticks that make up a game, but positional play is the architectural blueprint."