cutechess-cli question

Discussion of anything and everything relating to chess playing software and machines.

Moderators: hgm, Harvey Williamson, bob

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
lech
Posts: 1052
Joined: Sun Feb 14, 2010 9:02 pm

cutechess-cli question

Post by lech » Wed Nov 07, 2012 10:20 am

Why cutechess-cli doesn't allow to use only one engine in matches.
It would be an ideal tool to test changes in engines (same allocation in memory and hash). :D
Maybe, I can't be friendly, but let me be useful.

Joerg Oster
Posts: 684
Joined: Fri Mar 10, 2006 3:29 pm
Location: Germany

Re: cutechess-cli question

Post by Joerg Oster » Wed Nov 07, 2012 1:11 pm

Hi Marek,

do you want to play against the same engine?
Simply start the same engine twice.

If you want to play a gauntlet, one engine against several different opponents, then use "-tournament gauntlet". That's what I currently do.

Best,
Joerg.

P.S. I can't wait to see cutechess support pondering.
Jörg Oster

lech
Posts: 1052
Joined: Sun Feb 14, 2010 9:02 pm

Re: cutechess-cli question

Post by lech » Wed Nov 07, 2012 4:22 pm

Joerg Oster wrote:do you want to play against the same engine?
Simply start the same engine twice.
Hi Joerg,
Sometimes one of two loaded engines can play very weak by several (or more) games.
I think that it is much better to use only one allocation of memory (smaller error bar ?).
Of course the only one engine plays in two versions of some fragment of code.
It needs to add a code to distingush the versions.
best
Marek
Maybe, I can't be friendly, but let me be useful.

User avatar
ilari
Posts: 750
Joined: Mon Mar 27, 2006 5:45 pm
Location: Finland
Contact:

Re: cutechess-cli question

Post by ilari » Wed Nov 07, 2012 7:22 pm

lech wrote:
Joerg Oster wrote:do you want to play against the same engine?
Simply start the same engine twice.
Hi Joerg,
Sometimes one of two loaded engines can play very weak by several (or more) games.
I think that it is much better to use only one allocation of memory (smaller error bar ?).
Of course the only one engine plays in two versions of some fragment of code.
It needs to add a code to distingush the versions.
best
Marek
It's completely normal that a match between two identical engines can seem lopsided for several games or even a hundred games. It's just luck, which will even out if you play enough games. I don't see how using the same engine process to play both sides would help.

lech
Posts: 1052
Joined: Sun Feb 14, 2010 9:02 pm

Re: cutechess-cli question

Post by lech » Thu Nov 08, 2012 2:18 pm

ilari wrote:
lech wrote:
Joerg Oster wrote:do you want to play against the same engine?
Simply start the same engine twice.
Hi Joerg,
Sometimes one of two loaded engines can play very weak by several (or more) games.
I think that it is much better to use only one allocation of memory (smaller error bar ?).
Of course the only one engine plays in two versions of some fragment of code.
It needs to add a code to distingush the versions.
best
Marek
It's completely normal that a match between two identical engines can seem lopsided for several games or even a hundred games. It's just luck, which will even out if you play enough games. I don't see how using the same engine process to play both sides would help.
An engine is not a dice (rather Rubik's cube).
The second question is (if I may):
I try to test with option: st=x (time per move),
Why cutechess-cli returns <x loses on time>? I tested it at two original Stockfish-23-32-ja as opponents.
The third question is (if I may):
To get less random results it would be good to clear hash before each game. Can I do it?
Maybe, I can't be friendly, but let me be useful.

User avatar
lucasart
Posts: 3037
Joined: Mon May 31, 2010 11:29 am
Full name: lucasart
Contact:

Re: cutechess-cli question

Post by lucasart » Thu Nov 08, 2012 3:12 pm

lech wrote: To get less random results it would be good to clear hash before each game. Can I do it?
You need to program this into "your" engine. When the ucinewgame command is received, you need to clear the hash table. How do you expect cutechess-cli to clear your hash table for you ? cutechess-cli and your engine are separate processes, and cutechess-cli doesn't even know the adress in memory to clear, and even if it knew and tried to write on it, it would trigger a segmentation fault.

As for the idea of having only one engine, it doesn't make any sense whatsoever, IMO. Can you describe more precisely how you would want the experiment to work ?
Theory and practice sometimes clash. And when that happens, theory loses. Every single time.

mar
Posts: 1992
Joined: Fri Nov 26, 2010 1:00 pm
Location: Czech Republic
Full name: Martin Sedlak

Re: cutechess-cli question

Post by mar » Thu Nov 08, 2012 3:24 pm

lucasart wrote:How do you expect cutechess-cli to clear your hash table for you ?
What about reloading the engine? Of course this is not an option for bullet. Or sending clear hash command if it's UCI?

User avatar
lucasart
Posts: 3037
Joined: Mon May 31, 2010 11:29 am
Full name: lucasart
Contact:

Re: cutechess-cli question

Post by lucasart » Thu Nov 08, 2012 3:27 pm

mar wrote:Or sending clear hash command if it's UCI?
That's what I do in my engine. Although I defined an UCI option for that called AutoClearHash=true/false. If the option is true then ucinewgame clears the hash, otherwise it does nothing.
Is the "Clear Hash" (button) really specified by the UCI protocol ? If it is, then yes, cutechess-cli could send it.
Theory and practice sometimes clash. And when that happens, theory loses. Every single time.

lech
Posts: 1052
Joined: Sun Feb 14, 2010 9:02 pm

Re: cutechess-cli question

Post by lech » Thu Nov 08, 2012 3:32 pm

lucasart wrote:
lech wrote: To get less random results it would be good to clear hash before each game. Can I do it?
How do you expect cutechess-cli to clear your hash table for you ?
AFAK "Clear Hash" is an UCI (protocol) option and I can't understand why cutechess-cli is not able to send it? :?
Maybe, I can't be friendly, but let me be useful.

mar
Posts: 1992
Joined: Fri Nov 26, 2010 1:00 pm
Location: Czech Republic
Full name: Martin Sedlak

Re: cutechess-cli question

Post by mar » Thu Nov 08, 2012 3:40 pm

lucasart wrote: That's what I do in my engine. Although I defined an UCI option for that called AutoClearHash=true/false. If the option is true then ucinewgame clears the hash, otherwise it does nothing.
Is the "Clear Hash" (button) really specified by the UCI protocol ? If it is, then yes, cutechess-cli could send it.
I always clear hash whenever I receive ucinewgame, but I think that isn't mandatory.
Although there is "option name Clear Hash type button" mentioned in the UCI spec, I think it only serves as an example/recommendation so it's not part of the standard, which means you're right.
So the only compliant way would be to restart the engine, which would be terribly inefficient.

Post Reply