cutechess feature request: average search depth

[ilari]

Any news regarding added features or something?

Just to mention a few: huge performance improvements in parsing engine output (helps in fast games), infinite time control, "X time per move" time control, adjustable margin for the expiry of thinking time, custom engine options can be set in the engines.json config file, better PGN parser, support for multiline init strings, got rid of QJson library dependency, bug fixes, etc. Right now most of our work is going toward getting cutechess-gui to a usable and useful state.

BTW I have experienced somethng like this. I run a match of 100 games in 1 batch file with option recover added to one engine. This engine did indeed crashed, then it was successfully started, after the match I checked the pgn file (I did not set the debug option) I got 102 games, there were 2 games recorded prior to normal recovery. The first game with * result and the second game with 0-1 result in favor to the engine that did not crashed. The recorded 3rd game is ok till 102 games. Basically the official games is from 3rd to 102nd for a total of 100 games. Could it be possible for cutechess-cli to remove the 1st and 2nd game in this case? I actually ended up removing problematic engines to my sparring partner list. I was afraid that problematic engines and cutechess-cli will go on a crashed-recover-crashed-recover... loop.
Thanks for this wonderful tool.

Finally a bug report for v0.3.0! I have to admit that the -recover option wasn't tested thoroughly. You definitely found a bug, there should be only 100 games in the PGN file. It's probably going to be easy to fix, I'll let you know...

[ilari]

I recommend just looking at the first pgn file. Averaging 8 ply on the first move with 16 in the ending seems like it would be less informative than just opening the .pgn and saying aha! It's searching 8 in the opening.

-sam

Yes, this is a better way ! Thanks for the hint.

Ilari please ignore my request. Thanks.

Ok, I will
It sounded like a good idea, but yes, endgame search depths would distort the average too much.

[mcostalba]

Ok, I will
It sounded like a good idea, but yes, endgame search depths would distort the average too much.

Actually was not a good idea: to add a new option for a special case already resolved (and much better BTW) by the most general approach of reading the resulting pgn file.

Anyhow, speaking about features, the real big one that you still miss is to support tournaments among more then two engines....

[ilari]

Anyhow, speaking about features, the real big one that you still miss is to support tournaments among more then two engines....

It's on the agenda, but it's probably not going to be in the next release. The reason we're saving the best (tournaments) for last is that we first need to finish the engine configuration stuff, game management, etc.

[Engin]

i prefer to use LittleBlitzer because is easy to use and it do output some statistics of depth and some more.

at the moment its use only UCI protocol, but anyway i need only for UCI

the best is, i can run more games pararell at the same time on multicore hardwares

[BubbaTough]

Speaking of features, here are a couple that i would like, that may even be in there already in some form but i am failing to find.

1. Pgn start game. When supplying a pgn file of starting positions, i would like to be able to have the first game be from a number of my choice, not just the first pgn in the file. This is for restarting interrupted testing.

2. Random pgn game order. Instead of playing testing games in the order given in a pgn file of starting positions, i would like to be able to make the order random, but without repeating positions.

-sam

[Jan Brouwer]

2. Random pgn game order. Instead of playing testing games in the order given in a pgn file of starting positions, i would like to be able to make the order random, but without repeating positions.

-sam

+1 (shuffling)

Jan

[Houdini]

Finally a bug report for v0.3.0! I have to admit that the -recover option wasn't tested thoroughly. You definitely found a bug, there should be only 100 games in the PGN file. It's probably going to be easy to fix, I'll let you know...

Here's another bug report .

When using -concurrency and -wait options, Cutechess-cli often goes beyond the total number of games in a match. For example, when requesting 1000 games with -concurrency 8 and -wait 500 it often goes to 1003 or 1004 games. The higher the -wait value, the higher the overshoot generally is.
This behavior was also present in the previous version, it's not purely related to v0.3.

By the way, thanks for your very useful application!

Robert

[Ferdy]

i prefer to use LittleBlitzer because is easy to use and it do output some statistics of depth and some more.

at the moment its use only UCI protocol, but anyway i need only for UCI

the best is, i can run more games pararell at the same time on multicore hardwares

Cutechess-cli can also run more than 1 game using "-concurrency <n>" option, where n is the number of games to be run simultaneously, should not be more than number of cores or cpu.

[Ferdy]

Speaking of features, here are a couple that i would like, that may even be in there already in some form but i am failing to find.

1. Pgn start game. When supplying a pgn file of starting positions, i would like to be able to have the first game be from a number of my choice, not just the first pgn in the file. This is for restarting interrupted testing.

2. Random pgn game order. Instead of playing testing games in the order given in a pgn file of starting positions, i would like to be able to make the order random, but without repeating positions.

-sam

Item 1 - we might request to implement the capability to continue interrupted games once and for all.
I don't experience interrupted testing much, I have only experieced this when there was a brownout. Other than that I interrupt games when I don't like the result.
I usually run matches with 100 games in 1 batch command line. For fast TC I just re-run from start if there was iterruptions below 21st game, otherwise I will edit the test suite and run the batch again with the missing games.

Item 2 - any idea why testing has to be done in random without repetition?
Is this just the same if you run the complete test suite? say test suite have 2 games, so it's either test suite game 1 first or test suite game 2 first.
If you don't like to complete the whole test suite, then I recommend a tool from Frank's web site wherein it will generate a set of test suites from a given source pgn file. The tool will attempt to generate equal number of ECO openings depending on the availability in the source pgn file. If you have 1000 games in a source pgn file, the tool is capable of generating 1000/100 = 10 unique test suites with every test suites having 100 games inside.