cutechess feature request: average search depth

Discussion of anything and everything relating to chess playing software and machines.

Moderators: hgm, Harvey Williamson, bob

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
mcostalba
Posts: 2684
Joined: Sat Jun 14, 2008 7:17 pm

cutechess feature request: average search depth

Post by mcostalba » Sat Apr 02, 2011 1:00 pm

It is common for engine developers to use cutechess at very tight time controls so to accumulate many games in a not too long time. But sometime one pushes the pedal a bit too much and if TC is too aggressive then result could became not scalable at the commonly used longer TC.

So I would like to ask if it is possible to print out, after each match, not only the result bat also the average search depth of each of the opponents. This should be possible given that each engine normally sends iterative loop's depth. So that cutechess could track the ID depth reached just before sending bestmove and so be able to print statistic about searched depth after each game.

This would be very useful to correctly tune TC according to the kind of hardware available.

Thanks
Marco

Ralph Stoesser
Posts: 408
Joined: Sat Mar 06, 2010 8:28 am

Re: cutechess feature request: average search depth

Post by Ralph Stoesser » Sat Apr 02, 2011 1:26 pm

Agreed. It also would be nice to have an option to print out Elo information (diff, standard deviations, confidence) after every game.

User avatar
ilari
Posts: 750
Joined: Mon Mar 27, 2006 5:45 pm
Location: Finland
Contact:

Re: cutechess feature request: average search depth

Post by ilari » Sat Apr 02, 2011 3:33 pm

Sure, it's very doable, I'll add it to my TODO list. Cutechess-cli already includes the search depth in the PGN comment after each move.
It also would be nice to have an option to print out Elo information (diff, standard deviations, confidence) after every game.
Currently ELO difference is printed after the whole match. Maybe there could be a separate option for printing that kind of data after every game. I'll look into it.


We've committed a lot of code to our repository since the last release of cutechess-cli, so expect the next one to be a big one. You can also expect it to require more testing than most versions.

BubbaTough
Posts: 1154
Joined: Fri Jun 23, 2006 3:18 am

Re: cutechess feature request: average search depth

Post by BubbaTough » Sat Apr 02, 2011 3:45 pm

I recommend just looking at the first pgn file. Averaging 8 ply on the first move with 16 in the ending seems like it would be less informative than just opening the .pgn and saying aha! It's searching 8 in the opening.

-sam
mcostalba wrote:It is common for engine developers to use cutechess at very tight time controls so to accumulate many games in a not too long time. But sometime one pushes the pedal a bit too much and if TC is too aggressive then result could became not scalable at the commonly used longer TC.

So I would like to ask if it is possible to print out, after each match, not only the result bat also the average search depth of each of the opponents. This should be possible given that each engine normally sends iterative loop's depth. So that cutechess could track the ID depth reached just before sending bestmove and so be able to print statistic about searched depth after each game.

This would be very useful to correctly tune TC according to the kind of hardware available.

Thanks
Marco

Ferdy
Posts: 4071
Joined: Sun Aug 10, 2008 1:15 pm
Location: Philippines

Re: cutechess feature request: average search depth

Post by Ferdy » Sat Apr 02, 2011 4:02 pm

Ralph Stoesser wrote:Agreed. It also would be nice to have an option to print out Elo information (diff, standard deviations, confidence) after every game.
Any idea why elo info is necessary after every game? In my case I run bayeselo on the saved pgn after every match of (100 games) in a series of 10 matches or so. Inquering elo info after every game is too much for me :( .

This can be done in a batch file like the ff.
call match1
call match2
call check_elo

call match3
call match4
call check_elo

check_elo is a batch file that will run bayeselo and is able to
save elo info tables in a file for examination, or can be used (after processing) to decide whether to stop or continue the series of matches.

Ferdy
Posts: 4071
Joined: Sun Aug 10, 2008 1:15 pm
Location: Philippines

Re: cutechess feature request: average search depth

Post by Ferdy » Sat Apr 02, 2011 4:31 pm

mcostalba wrote:It is common for engine developers to use cutechess at very tight time controls so to accumulate many games in a not too long time. But sometime one pushes the pedal a bit too much and if TC is too aggressive then result could became not scalable at the commonly used longer TC.

So I would like to ask if it is possible to print out, after each match, not only the result bat also the average search depth of each of the opponents. This should be possible given that each engine normally sends iterative loop's depth. So that cutechess could track the ID depth reached just before sending bestmove and so be able to print statistic about searched depth after each game.

This would be very useful to correctly tune TC according to the kind of hardware available.

Thanks
Marco
With different hardwares I solved this problem by running different test suites in every pc, pc-1 will run test-1, pc-2 will run test-2, ... then collect games for bayeselo input. This is even more interesting since you can run sicilian test suite in pc-1, ruy lopez test suite in pc-2 and so on, after which you will be able to tell if the changes you have made for version xxx is good in the sicilian :) .

Another solution is to run bench command in every hardware and to finally allocate time control in every hardware based on the bench results similar I think what CCRL is doing.

Ralph Stoesser
Posts: 408
Joined: Sat Mar 06, 2010 8:28 am

Re: cutechess feature request: average search depth

Post by Ralph Stoesser » Sat Apr 02, 2011 5:09 pm

Ferdy wrote:
Ralph Stoesser wrote:Agreed. It also would be nice to have an option to print out Elo information (diff, standard deviations, confidence) after every game.
Any idea why elo info is necessary after every game? In my case I run bayeselo on the saved pgn after every match of (100 games) in a series of 10 matches or so. Inquering elo info after every game is too much for me :( .

This can be done in a batch file like the ff.
call match1
call match2
call check_elo

call match3
call match4
call check_elo

check_elo is a batch file that will run bayeselo and is able to
save elo info tables in a file for examination, or can be used (after processing) to decide whether to stop or continue the series of matches.
Nice, but this would be too sophisticated. Batch file, shell script??? We like it dumb and easy.
iPod generation regards ;-)

Daniel Shawul
Posts: 3746
Joined: Tue Mar 14, 2006 10:34 am
Location: Ethiopia
Contact:

Re: cutechess feature request: average search depth

Post by Daniel Shawul » Sat Apr 02, 2011 5:25 pm

That is what I do too . On a cluster, every 3 minutes or so bayeselo is called on the PGN file. So I can decide and stop the tourney if it is doing very bad.
Doing it after every game is an overkill. I don't know if it is possible to do that when testing on a cluster, and honestly don't see the point either...

Ferdy
Posts: 4071
Joined: Sun Aug 10, 2008 1:15 pm
Location: Philippines

Re: cutechess feature request: average search depth

Post by Ferdy » Sat Apr 02, 2011 5:30 pm

ilari wrote:Sure, it's very doable, I'll add it to my TODO list. Cutechess-cli already includes the search depth in the PGN comment after each move.
It also would be nice to have an option to print out Elo information (diff, standard deviations, confidence) after every game.
Currently ELO difference is printed after the whole match. Maybe there could be a separate option for printing that kind of data after every game. I'll look into it.


We've committed a lot of code to our repository since the last release of cutechess-cli, so expect the next one to be a big one. You can also expect it to require more testing than most versions.
Any news regarding added features or something?
BTW I have experienced somethng like this. I run a match of 100 games in 1 batch file with option recover added to one engine. This engine did indeed crashed, then it was successfully started, after the match I checked the pgn file (I did not set the debug option) I got 102 games, there were 2 games recorded prior to normal recovery. The first game with * result and the second game with 0-1 result in favor to the engine that did not crashed. The recorded 3rd game is ok till 102 games. Basically the official games is from 3rd to 102nd for a total of 100 games. Could it be possible for cutechess-cli to remove the 1st and 2nd game in this case? I actually ended up removing problematic engines to my sparring partner list. I was afraid that problematic engines and cutechess-cli will go on a crashed-recover-crashed-recover... loop. :(
Thanks for this wonderful tool.

mcostalba
Posts: 2684
Joined: Sat Jun 14, 2008 7:17 pm

Re: cutechess feature request: average search depth

Post by mcostalba » Sat Apr 02, 2011 6:06 pm

BubbaTough wrote:I recommend just looking at the first pgn file. Averaging 8 ply on the first move with 16 in the ending seems like it would be less informative than just opening the .pgn and saying aha! It's searching 8 in the opening.

-sam
Yes, this is a better way ! Thanks for the hint.

Ilari please ignore my request. Thanks.

Post Reply