cutechess-cli question

[lech]

Why cutechess-cli doesn't allow to use only one engine in matches.
It would be an ideal tool to test changes in engines (same allocation in memory and hash).

[Joerg Oster]

Hi Marek,

do you want to play against the same engine?
Simply start the same engine twice.

If you want to play a gauntlet, one engine against several different opponents, then use "-tournament gauntlet". That's what I currently do.

Best,
Joerg.

P.S. I can't wait to see cutechess support pondering.

[lech]

do you want to play against the same engine?
Simply start the same engine twice.

Hi Joerg,
Sometimes one of two loaded engines can play very weak by several (or more) games.
I think that it is much better to use only one allocation of memory (smaller error bar ?).
Of course the only one engine plays in two versions of some fragment of code.
It needs to add a code to distingush the versions.
best
Marek

[ilari]

do you want to play against the same engine?
Simply start the same engine twice.

Hi Joerg,
Sometimes one of two loaded engines can play very weak by several (or more) games.
I think that it is much better to use only one allocation of memory (smaller error bar ?).
Of course the only one engine plays in two versions of some fragment of code.
It needs to add a code to distingush the versions.
best
Marek

It's completely normal that a match between two identical engines can seem lopsided for several games or even a hundred games. It's just luck, which will even out if you play enough games. I don't see how using the same engine process to play both sides would help.

[lech]

do you want to play against the same engine?
Simply start the same engine twice.

Hi Joerg,
Sometimes one of two loaded engines can play very weak by several (or more) games.
I think that it is much better to use only one allocation of memory (smaller error bar ?).
Of course the only one engine plays in two versions of some fragment of code.
It needs to add a code to distingush the versions.
best
Marek

It's completely normal that a match between two identical engines can seem lopsided for several games or even a hundred games. It's just luck, which will even out if you play enough games. I don't see how using the same engine process to play both sides would help.

An engine is not a dice (rather Rubik's cube).
The second question is (if I may):
I try to test with option: st=x (time per move),
Why cutechess-cli returns <x loses on time>? I tested it at two original Stockfish-23-32-ja as opponents.
The third question is (if I may):
To get less random results it would be good to clear hash before each game. Can I do it?

[lucasart]

To get less random results it would be good to clear hash before each game. Can I do it?

You need to program this into "your" engine. When the ucinewgame command is received, you need to clear the hash table. How do you expect cutechess-cli to clear your hash table for you ? cutechess-cli and your engine are separate processes, and cutechess-cli doesn't even know the adress in memory to clear, and even if it knew and tried to write on it, it would trigger a segmentation fault.

As for the idea of having only one engine, it doesn't make any sense whatsoever, IMO. Can you describe more precisely how you would want the experiment to work ?

[mar]

How do you expect cutechess-cli to clear your hash table for you ?

What about reloading the engine? Of course this is not an option for bullet. Or sending clear hash command if it's UCI?

[lucasart]

Or sending clear hash command if it's UCI?

That's what I do in my engine. Although I defined an UCI option for that called AutoClearHash=true/false. If the option is true then ucinewgame clears the hash, otherwise it does nothing.
Is the "Clear Hash" (button) really specified by the UCI protocol ? If it is, then yes, cutechess-cli could send it.

[lech]

To get less random results it would be good to clear hash before each game. Can I do it?

How do you expect cutechess-cli to clear your hash table for you ?

AFAK "Clear Hash" is an UCI (protocol) option and I can't understand why cutechess-cli is not able to send it?

[mar]

That's what I do in my engine. Although I defined an UCI option for that called AutoClearHash=true/false. If the option is true then ucinewgame clears the hash, otherwise it does nothing.
Is the "Clear Hash" (button) really specified by the UCI protocol ? If it is, then yes, cutechess-cli could send it.

I always clear hash whenever I receive ucinewgame, but I think that isn't mandatory.
Although there is "option name Clear Hash type button" mentioned in the UCI spec, I think it only serves as an example/recommendation so it's not part of the standard, which means you're right.
So the only compliant way would be to restart the engine, which would be terribly inefficient.