Why cutechess-cli doesn't allow to use only one engine in matches.
It would be an ideal tool to test changes in engines (same allocation in memory and hash).
cutechess-cli question
Moderators: hgm, Rebel, chrisw
-
- Posts: 1136
- Joined: Sun Feb 14, 2010 10:02 pm
cutechess-cli question
Maybe, I can't be friendly, but let me be useful.
-
- Posts: 937
- Joined: Fri Mar 10, 2006 4:29 pm
- Location: Germany
Re: cutechess-cli question
Hi Marek,
do you want to play against the same engine?
Simply start the same engine twice.
If you want to play a gauntlet, one engine against several different opponents, then use "-tournament gauntlet". That's what I currently do.
Best,
Joerg.
P.S. I can't wait to see cutechess support pondering.
do you want to play against the same engine?
Simply start the same engine twice.
If you want to play a gauntlet, one engine against several different opponents, then use "-tournament gauntlet". That's what I currently do.
Best,
Joerg.
P.S. I can't wait to see cutechess support pondering.
Jörg Oster
-
- Posts: 1136
- Joined: Sun Feb 14, 2010 10:02 pm
Re: cutechess-cli question
Hi Joerg,Joerg Oster wrote:do you want to play against the same engine?
Simply start the same engine twice.
Sometimes one of two loaded engines can play very weak by several (or more) games.
I think that it is much better to use only one allocation of memory (smaller error bar ?).
Of course the only one engine plays in two versions of some fragment of code.
It needs to add a code to distingush the versions.
best
Marek
Maybe, I can't be friendly, but let me be useful.
-
- Posts: 750
- Joined: Mon Mar 27, 2006 7:45 pm
- Location: Finland
Re: cutechess-cli question
It's completely normal that a match between two identical engines can seem lopsided for several games or even a hundred games. It's just luck, which will even out if you play enough games. I don't see how using the same engine process to play both sides would help.lech wrote:Hi Joerg,Joerg Oster wrote:do you want to play against the same engine?
Simply start the same engine twice.
Sometimes one of two loaded engines can play very weak by several (or more) games.
I think that it is much better to use only one allocation of memory (smaller error bar ?).
Of course the only one engine plays in two versions of some fragment of code.
It needs to add a code to distingush the versions.
best
Marek
-
- Posts: 1136
- Joined: Sun Feb 14, 2010 10:02 pm
Re: cutechess-cli question
An engine is not a dice (rather Rubik's cube).ilari wrote:It's completely normal that a match between two identical engines can seem lopsided for several games or even a hundred games. It's just luck, which will even out if you play enough games. I don't see how using the same engine process to play both sides would help.lech wrote:Hi Joerg,Joerg Oster wrote:do you want to play against the same engine?
Simply start the same engine twice.
Sometimes one of two loaded engines can play very weak by several (or more) games.
I think that it is much better to use only one allocation of memory (smaller error bar ?).
Of course the only one engine plays in two versions of some fragment of code.
It needs to add a code to distingush the versions.
best
Marek
The second question is (if I may):
I try to test with option: st=x (time per move),
Why cutechess-cli returns <x loses on time>? I tested it at two original Stockfish-23-32-ja as opponents.
The third question is (if I may):
To get less random results it would be good to clear hash before each game. Can I do it?
Maybe, I can't be friendly, but let me be useful.
-
- Posts: 3232
- Joined: Mon May 31, 2010 1:29 pm
- Full name: lucasart
Re: cutechess-cli question
You need to program this into "your" engine. When the ucinewgame command is received, you need to clear the hash table. How do you expect cutechess-cli to clear your hash table for you ? cutechess-cli and your engine are separate processes, and cutechess-cli doesn't even know the adress in memory to clear, and even if it knew and tried to write on it, it would trigger a segmentation fault.lech wrote: To get less random results it would be good to clear hash before each game. Can I do it?
As for the idea of having only one engine, it doesn't make any sense whatsoever, IMO. Can you describe more precisely how you would want the experiment to work ?
Theory and practice sometimes clash. And when that happens, theory loses. Every single time.
-
- Posts: 2554
- Joined: Fri Nov 26, 2010 2:00 pm
- Location: Czech Republic
- Full name: Martin Sedlak
Re: cutechess-cli question
What about reloading the engine? Of course this is not an option for bullet. Or sending clear hash command if it's UCI?lucasart wrote:How do you expect cutechess-cli to clear your hash table for you ?
-
- Posts: 3232
- Joined: Mon May 31, 2010 1:29 pm
- Full name: lucasart
Re: cutechess-cli question
That's what I do in my engine. Although I defined an UCI option for that called AutoClearHash=true/false. If the option is true then ucinewgame clears the hash, otherwise it does nothing.mar wrote:Or sending clear hash command if it's UCI?
Is the "Clear Hash" (button) really specified by the UCI protocol ? If it is, then yes, cutechess-cli could send it.
Theory and practice sometimes clash. And when that happens, theory loses. Every single time.
-
- Posts: 1136
- Joined: Sun Feb 14, 2010 10:02 pm
Re: cutechess-cli question
AFAK "Clear Hash" is an UCI (protocol) option and I can't understand why cutechess-cli is not able to send it?lucasart wrote:How do you expect cutechess-cli to clear your hash table for you ?lech wrote: To get less random results it would be good to clear hash before each game. Can I do it?
Maybe, I can't be friendly, but let me be useful.
-
- Posts: 2554
- Joined: Fri Nov 26, 2010 2:00 pm
- Location: Czech Republic
- Full name: Martin Sedlak
Re: cutechess-cli question
I always clear hash whenever I receive ucinewgame, but I think that isn't mandatory.lucasart wrote: That's what I do in my engine. Although I defined an UCI option for that called AutoClearHash=true/false. If the option is true then ucinewgame clears the hash, otherwise it does nothing.
Is the "Clear Hash" (button) really specified by the UCI protocol ? If it is, then yes, cutechess-cli could send it.
Although there is "option name Clear Hash type button" mentioned in the UCI spec, I think it only serves as an example/recommendation so it's not part of the standard, which means you're right.
So the only compliant way would be to restart the engine, which would be terribly inefficient.