Setting an average time per position for long batch analysis

[gordonr]

Hi,

Suppose I have a set of unrelated positions and a long time period to analyse the whole set. I could divide the overall time period into equal time slots and analyse each position for a fixed time. However, this doesn’t sound good. Firstly, I’d often be terminating a search in the middle of an iteration rather than letting that iteration complete. I believe this is to be avoided unless the iteration is taking far too long. Secondly, I should use the engine’s logic to help identify positions that should be analysed for longer or shorter than average (e.g. forcing moves).

One use case is a set of correspondence game positions to be analysed while I’m on holiday for a week or so. I don’t care if the analysis completes a bit sooner or later than expected, as long as it’s not too far out. So it’s fine for a week’s worth of analysis to take between 5 to 9 days, but 2 or 12 days is missing the mark too much, for example.

I’m familiar with the UCI protocol to some extent and have Java code for invoking analysis. What’s the best way for me to try to aim for this “average time per position”? Should I experiment with setting a "sudden death" time control of e.g. 2 days for each position and seeing how that averages out, adjusting as necessary? Maybe this will vary significantly across engines and I’ll need a different time control for each engine in order to aim for the same “first move” average. I’m using Stockfish, Komodo and Houdini.

I’ve not used any batch analysis in any Chessbase products or Aquarium, etc. so I don’t know what they support or do.

thanks

Gordon

[hgm]

In WinBoard the EPD mode uses the same time controls as are also used for games. So you can set fixed time per move, which indeed has the disadvantage of cutting iterations short, or you can set 40 moves per time T, so that it uses on average T/40 on each position, but tries to tweek the exact time so not much is wasted on incomplete iterations. Of course as soon as the engine moves, it goes to the next position in the EPD file.

[yurikvelo]

What is your purpose?
Do you need only best move, or you need as deep value as possible?

E.g. at N=1000 MN
PV1: +0.42
PV2: -5.45

There is almost zero chance for PV1 to become not best move at N=10 000 MN, but it is highly probable value won't be +0.42

[gordonr]

you can set 40 moves per time T

Thanks. I'll experiment with that.

[gordonr]

What is your purpose?
Do you need only best move, or you need as deep value as possible?

E.g. at N=1000 MN
PV1: +0.42
PV2: -5.45

There is almost zero chance for PV1 to become not best move at N=10 000 MN, but it is highly probable value won't be +0.42

I want a best move along an evaluation which is as accurate as possible. You raise an interesting point about comparing the top two PVs but I'm wondering how often the delta is so clear cut. I believe some top engines implement some form of quick move. I don't know the logic but I guess it's something clever enough that I hope to avoid trying to emulate myself.

Thanks.

[hgm]

you can set 40 moves per time T

Thanks. I'll experiment with that.

Note you will need to use at least version 4.9.0 for the -epd option to be recognized. This was never officially released as a binary, but the http://hgm.nubati.net/WinBoard-AA.zip package contains a beta version of 4.9.2.

Also beware that WinBoard's standard EPD reporting might be too limited for your purpose. But you can work around that by having it create a winboard.debug file with the addifitional option -debug. This file will contain all engine-GUI communication, so that you can see scores and PVs for every depth the engine calculated afterwards.

[yurikvelo]

I want a best move along an evaluation which is as accurate as possible.

If you increase accuracy in one position, you sacrifice accuracy and possibly better PV in other position.

You raise an interesting point about comparing the top two PVs but I'm wondering how often the delta is so clear cut.

Example is exaggerated, but it should make your answer more clear: you need as deep numerical evaluation as possible (ideally go beyond +-2.5 or 0.00, so the game outcome could be declared as win/draw/loss), or you need just best PV, without resolving game outcome?

I believe some top engines implement some form of quick move. I don't know the logic but I guess it's something clever enough that I hope to avoid trying to emulate myself.

Top engines time management try to answer question: what is the probability for best PV to change, if I invest N+1 more time in this position.
They analyze both distance between 1 and 2nd PV and PV-stability. If distance is stable, PV do not shuffle and distance to 2nd is high enough - there is low chance of fail-low for 1st PV with more Depth.
PV stability and PV distance somehow are compared - more distance allow more fluctuations to keep probability low.

If you save 24 hours in position with "clear better 1st PV", you can double time for "shuffling position"

Couple examples of graphs for long analysis here: http://talkchess.com/forum/viewtopic.ph ... 3&start=10

If you run analysis, and set minimum amount of 10 000 MN (example), and reach for example D=39. If from D=20 to D=39 1st line was always 20% away from 2nd line, chance for fail-low with 20x more time (200 000 MN) is very low.

[gordonr]

Thanks Muller and Yuri for your help.

Many of the positions I analyse do not have a clear best move and aren't close to being resolved. My main aim is to spend the engine's overall time better by not allocating an equal amount of time to every position but instead using the engine's time management.

I need as reliable an evaluation as possible in case I want to later compare other lines. For example, I may try some ideas of my own and see how the evals compare.

[Dann Corbit]

It depends on your goals.
I like to analyze at 37 plies.
If I find a really interesting position, I analyze deeper.
If I know an answer is wrong (due to statistics) I analyze it deeper.

To me, the most interesting position is one that results in a computer novelty which greatly outperforms the statistical outcome.

They are fairly rare, but I have some thousands of them.