cutechess feature request: average search depth

[mcostalba]

It is common for engine developers to use cutechess at very tight time controls so to accumulate many games in a not too long time. But sometime one pushes the pedal a bit too much and if TC is too aggressive then result could became not scalable at the commonly used longer TC.

So I would like to ask if it is possible to print out, after each match, not only the result bat also the average search depth of each of the opponents. This should be possible given that each engine normally sends iterative loop's depth. So that cutechess could track the ID depth reached just before sending bestmove and so be able to print statistic about searched depth after each game.

This would be very useful to correctly tune TC according to the kind of hardware available.

Thanks
Marco

[Ralph Stoesser]

Agreed. It also would be nice to have an option to print out Elo information (diff, standard deviations, confidence) after every game.

[ilari]

Sure, it's very doable, I'll add it to my TODO list. Cutechess-cli already includes the search depth in the PGN comment after each move.

It also would be nice to have an option to print out Elo information (diff, standard deviations, confidence) after every game.

Currently ELO difference is printed after the whole match. Maybe there could be a separate option for printing that kind of data after every game. I'll look into it.


We've committed a lot of code to our repository since the last release of cutechess-cli, so expect the next one to be a big one. You can also expect it to require more testing than most versions.

[BubbaTough]

I recommend just looking at the first pgn file. Averaging 8 ply on the first move with 16 in the ending seems like it would be less informative than just opening the .pgn and saying aha! It's searching 8 in the opening.

-sam

It is common for engine developers to use cutechess at very tight time controls so to accumulate many games in a not too long time. But sometime one pushes the pedal a bit too much and if TC is too aggressive then result could became not scalable at the commonly used longer TC.

So I would like to ask if it is possible to print out, after each match, not only the result bat also the average search depth of each of the opponents. This should be possible given that each engine normally sends iterative loop's depth. So that cutechess could track the ID depth reached just before sending bestmove and so be able to print statistic about searched depth after each game.

This would be very useful to correctly tune TC according to the kind of hardware available.

Thanks
Marco

[Ferdy]

Agreed. It also would be nice to have an option to print out Elo information (diff, standard deviations, confidence) after every game.

Any idea why elo info is necessary after every game? In my case I run bayeselo on the saved pgn after every match of (100 games) in a series of 10 matches or so. Inquering elo info after every game is too much for me .

This can be done in a batch file like the ff.
call match1
call match2
call check_elo

call match3
call match4
call check_elo

check_elo is a batch file that will run bayeselo and is able to
save elo info tables in a file for examination, or can be used (after processing) to decide whether to stop or continue the series of matches.

[Ferdy]

It is common for engine developers to use cutechess at very tight time controls so to accumulate many games in a not too long time. But sometime one pushes the pedal a bit too much and if TC is too aggressive then result could became not scalable at the commonly used longer TC.

So I would like to ask if it is possible to print out, after each match, not only the result bat also the average search depth of each of the opponents. This should be possible given that each engine normally sends iterative loop's depth. So that cutechess could track the ID depth reached just before sending bestmove and so be able to print statistic about searched depth after each game.

This would be very useful to correctly tune TC according to the kind of hardware available.

Thanks
Marco

With different hardwares I solved this problem by running different test suites in every pc, pc-1 will run test-1, pc-2 will run test-2, ... then collect games for bayeselo input. This is even more interesting since you can run sicilian test suite in pc-1, ruy lopez test suite in pc-2 and so on, after which you will be able to tell if the changes you have made for version xxx is good in the sicilian .

Another solution is to run bench command in every hardware and to finally allocate time control in every hardware based on the bench results similar I think what CCRL is doing.

[Ralph Stoesser]

Agreed. It also would be nice to have an option to print out Elo information (diff, standard deviations, confidence) after every game.

Any idea why elo info is necessary after every game? In my case I run bayeselo on the saved pgn after every match of (100 games) in a series of 10 matches or so. Inquering elo info after every game is too much for me .

This can be done in a batch file like the ff.
call match1
call match2
call check_elo

call match3
call match4
call check_elo

check_elo is a batch file that will run bayeselo and is able to
save elo info tables in a file for examination, or can be used (after processing) to decide whether to stop or continue the series of matches.

Nice, but this would be too sophisticated. Batch file, shell script??? We like it dumb and easy.
iPod generation regards

[Daniel Shawul]

That is what I do too . On a cluster, every 3 minutes or so bayeselo is called on the PGN file. So I can decide and stop the tourney if it is doing very bad.
Doing it after every game is an overkill. I don't know if it is possible to do that when testing on a cluster, and honestly don't see the point either...

[Ferdy]

Sure, it's very doable, I'll add it to my TODO list. Cutechess-cli already includes the search depth in the PGN comment after each move.

It also would be nice to have an option to print out Elo information (diff, standard deviations, confidence) after every game.

Currently ELO difference is printed after the whole match. Maybe there could be a separate option for printing that kind of data after every game. I'll look into it.


We've committed a lot of code to our repository since the last release of cutechess-cli, so expect the next one to be a big one. You can also expect it to require more testing than most versions.

Any news regarding added features or something?
BTW I have experienced somethng like this. I run a match of 100 games in 1 batch file with option recover added to one engine. This engine did indeed crashed, then it was successfully started, after the match I checked the pgn file (I did not set the debug option) I got 102 games, there were 2 games recorded prior to normal recovery. The first game with * result and the second game with 0-1 result in favor to the engine that did not crashed. The recorded 3rd game is ok till 102 games. Basically the official games is from 3rd to 102nd for a total of 100 games. Could it be possible for cutechess-cli to remove the 1st and 2nd game in this case? I actually ended up removing problematic engines to my sparring partner list. I was afraid that problematic engines and cutechess-cli will go on a crashed-recover-crashed-recover... loop.
Thanks for this wonderful tool.

[mcostalba]

I recommend just looking at the first pgn file. Averaging 8 ply on the first move with 16 in the ending seems like it would be less informative than just opening the .pgn and saying aha! It's searching 8 in the opening.

-sam

Yes, this is a better way ! Thanks for the hint.

Ilari please ignore my request. Thanks.