How much elo is pondering worth?

[bob]

Has this question been researched?

The basic idea is saving time.

I just looked at one game played against houdini on ICC. Crafty correctly predicted it's opponent's move almost 60% of the time.

If you assume an equal distribution, about 1/2 of those moves will be made in zero time since crafty predicted correctly and searched while the opponent was searching. It the other half, even though it had predicted, so had its opponent and it was making instant moves.

So 30% of the moves are free. Adding about 30% extra time on your clock. That is probably in the 20-30 Elo range. Maybe more or less depending on how the search allocates time.

[bob]

I have one scant piece of data related to your question. Playing 750 games each at the CCRL blitz time control against a gauntlet of opponents, Gaviota 0.86 with ponder on measured 66 Elo stronger than ponder off (as measured by Ordo).

Was this on a dual-core box so that the pondering doesn't hurt the opponent?

[michiguel]

Thank you!

66 elo is a lot. You would think that pondering at most doubles your
thinking time. So 70 elo should be an upperbound.

It appears that Gaviota reaches this upperbound in your experiment but of course there are error bars to be considered which are quite large for
750 games.

If 70 points is the doubling, isn't pondering worth significantly less? The ponder hits are about 60% for unrelated engines, so I would expect ~45 points if 70 is the doubling.

That assumes that 40% of the time, when the prediction was wrong, the time was wasted. It is possible that it was not completely useless. In other words, searching a wrong move could still fill up the hashtable with useful information. Sometimes the prediction is wrong, but it will lead to a transposition later.

Miguel

[michiguel]

Has this question been researched?

The basic idea is saving time.

I just looked at one game played against houdini on ICC. Crafty correctly predicted it's opponent's move almost 60% of the time.

If you assume an equal distribution, about 1/2 of those moves will be made in zero time since crafty predicted correctly and searched while the opponent was searching. It the other half, even though it had predicted, so had its opponent and it was making instant moves.

So 30% of the moves are free. Adding about 30% extra time on your clock. That is probably in the 20-30 Elo range. Maybe more or less depending on how the search allocates time.

You are not taking into account your opponent wins 20-30, so that is 40-60. If you do not ponder, you lose 30%, but your opponent wins 30% because now his efficiency goes from 30 to 60 in you scenario.

Miguel

[Adam Hair]

Quad core. Two of the cores were not being used during the ponder on test.

The ponder off test was conducted using cutechess with concurrency = 3. So three matches were running at a time.

[Laskos]

Thank you!

66 elo is a lot. You would think that pondering at most doubles your
thinking time. So 70 elo should be an upperbound.

It appears that Gaviota reaches this upperbound in your experiment but of course there are error bars to be considered which are quite large for
750 games.

If 70 points is the doubling, isn't pondering worth significantly less? The ponder hits are about 60% for unrelated engines, so I would expect ~45 points if 70 is the doubling.

That assumes that 40% of the time, when the prediction was wrong, the time was wasted. It is possible that it was not completely useless. In other words, searching a wrong move could still fill up the hashtable with useful information. Sometimes the prediction is wrong, but it will lead to a transposition later.

Miguel

The efficiency of these 40% could not be higher than 20% or so, for a total of 60%+8%=68%, or 50 Elo points if doubling is 70 points.

[bob]

Quad core. Two of the cores were not being used during the ponder on test.

The ponder off test was conducted using cutechess with concurrency = 3. So three matches were running at a time.

That's a big gain, unless we are talking very fast time controls. I typically see 50-70 for doubling the speed or time per move. Pondering can't consistently give even a 50% speedup, because of the way the two programs get into a ponder-hit-ponder-hit cycle, and SOMEBODY has to think for 3 minutes or whatever.

Realistically, the last time I measured this, the time savings averaged about 30% over a 30K match on my cluster. That's probably close to 20 Elo or a bit more, at least for my code.

[bob]

Has this question been researched?

The basic idea is saving time.

I just looked at one game played against houdini on ICC. Crafty correctly predicted it's opponent's move almost 60% of the time.

If you assume an equal distribution, about 1/2 of those moves will be made in zero time since crafty predicted correctly and searched while the opponent was searching. It the other half, even though it had predicted, so had its opponent and it was making instant moves.

So 30% of the moves are free. Adding about 30% extra time on your clock. That is probably in the 20-30 Elo range. Maybe more or less depending on how the search allocates time.

You are not taking into account your opponent wins 20-30, so that is 40-60. If you do not ponder, you lose 30%, but your opponent wins 30% because now his efficiency goes from 30 to 60 in you scenario.

Miguel

I am thinking of A vs B.

1. No pondering, Elo is equal for both.

2. A ponders, B does not. A gains 30 Elo due to that.

3. A does not, B does. B gains 30 Elo.

4. Both ponder, back to zero.

It sounds like you are talking A ponders and B doesn't, against B ponders and A does not. There I agree it would be doubled...

[bob]

Thank you!

66 elo is a lot. You would think that pondering at most doubles your
thinking time. So 70 elo should be an upperbound.

It appears that Gaviota reaches this upperbound in your experiment but of course there are error bars to be considered which are quite large for
750 games.

If 70 points is the doubling, isn't pondering worth significantly less? The ponder hits are about 60% for unrelated engines, so I would expect ~45 points if 70 is the doubling.

That assumes that 40% of the time, when the prediction was wrong, the time was wasted. It is possible that it was not completely useless. In other words, searching a wrong move could still fill up the hashtable with useful information. Sometimes the prediction is wrong, but it will lead to a transposition later.

Miguel

That's a point worth considering. Although, I must say, I am generally time-based and will search the same time either way. But the search could be of higher quality. I'm trying to think of a way to measure this but the two parts are so intertwined, nothing directly comes to mind.

[Adam Hair]

Quad core. Two of the cores were not being used during the ponder on test.

The ponder off test was conducted using cutechess with concurrency = 3. So three matches were running at a time.

That's a big gain, unless we are talking very fast time controls. I typically see 50-70 for doubling the speed or time per move. Pondering can't consistently give even a 50% speedup, because of the way the two programs get into a ponder-hit-ponder-hit cycle, and SOMEBODY has to think for 3 minutes or whatever.

Realistically, the last time I measured this, the time savings averaged about 30% over a 30K match on my cluster. That's probably close to 20 Elo or a bit more, at least for my code.

The ponder off games were played for the CCRL 40/4 (40 moves in 4 minutes on the CCRL reference computer) database. For my computer, the roughly equivalent time control is 40 moves in 155 seconds.

A few weeks later, there was some discussion at TCEC about whether using all cores with ponder off produced higher quality games than using half the cores with ponder on. I decided to see how much better Gaviota performed on one core while pondering as compared to one core and ponder off. I compared the ponder on games against the ponder off games, using the same opponents and the same positions.

On my computer (based on several experiments), the strength increase when doubling the speed at 40/155 seconds is somewhere around 70 to 90 Elo. This roughly agrees with what other people have found.

66 Elo does seem a bit high. However, only 750 games were played, so the true gain for Gaviota on my computer at that time control could be more like 45 Elo.

There is another piece of data that supports my result. Frank Quisinsky (google SWCR chess) created a ponder on rating list a couple of years ago, using a time control and processor that was roughly equivalent to 40/15 CEGT (so longer than my games). He did an experiment with Crafty 23.3 64bit, playing games with ponder off in addition to the ponder on games. I am not sure if he used exactly the same conditions (same opponents and same openings), but he measured a gain of 46 Elo.