In the aspiration window framework the engine finds a new, better move (e3h6) but the time is not enough to establish the score and the best move is not updated. It plays finally the old best move.
In the aspiration window framework the engine finds a new, better move (e3h6) but the time is not enough to establish the score and the best move is not updated. It plays finally the old best move.
Yes, this is a bug I am aware of. But any time you see something like this let me know and I will try to fix it for next version.
Capital punishment would be more effective as a preventive measure if it were administered prior to the crime.
Laszlo Gaspar wrote:In the aspiration window framework the engine finds a new, better move (e3h6) but the time is not enough to establish the score and the best move is not updated. It plays finally the old best move.
Some months ago I made some changes to the root search of my engine and the engine suddenly got 30-35 elo weaker when I played it against the old version. I undid most of the changes to get back as close as possible to the old version. In the end, node counts were exactly the same as those of the old version, but the 30-35 elo gap remained.
Guess what the bug was
I suppose this bug somehow hurts Komodo CCT less than it hurt my engine, or maybe self-play magnified its effect. (Or maybe it is only under certain conditions that Komodo fails to play the move that failed high but for which a score has not yet been established?)
In the aspiration window framework the engine finds a new, better move (e3h6) but the time is not enough to establish the score and the best move is not updated. It plays finally the old best move.
It is possible this a major bug - but I doubt it will have much impact on the strength. Still, I should probably check it out sooner rather than later.
The reason I don't expect it to be major is that once an iteration has started, Komodo is going to try really hard to complete it. So when I fix it I will also do the UCI fail hi and fail low reported the proper way.
I have to wonder how many games Komodo failed to win or draw due to this bug.
Capital punishment would be more effective as a preventive measure if it were administered prior to the crime.
Don wrote:The reason I don't expect it to be major is that once an iteration has started, Komodo is going to try really hard to complete it.
Ah, good point. This must be the main reason why it does not affect Komodo as badly as it did my engine. (Obviously the reason for my assumption that it does not affect Komodo so badly is that even with this bug Komodo is playing quite well )
In the aspiration window framework the engine finds a new, better move (e3h6) but the time is not enough to establish the score and the best move is not updated. It plays finally the old best move.
It is possible this a major bug - but I doubt it will have much impact on the strength. Still, I should probably check it out sooner rather than later.
The reason I don't expect it to be major is that once an iteration has started, Komodo is going to try really hard to complete it. So when I fix it I will also do the UCI fail hi and fail low reported the proper way.
I have to wonder how many games Komodo failed to win or draw due to this bug.
I did some research with Critter about handling fail highs at root. I measured the probability of unresolved fail highs being the new best move. Whenever the time management decided to stop the search and there was an unresolved fail high, I forced the engine to finish the iteration and logged the result to see if it really found a better move.
- if a move failed high ONCE, there was an 88% probability that this will be the new best move (this surprised me - I expected a higher number)
- if a move failed high TWICE, the probability increased to 97.5%
- on THIRD fail high the probability was 99.97%
I found out that if the time is really running out, it is quite safe to accept a new move after it failed high 2x. Obviously, if there is enough time, the safest way is to finish the iteration.
rvida wrote:
I did some research with Critter about handling fail highs at root. I measured the probability of unresolved fail highs being the new best move. Whenever the time management decided to stop the search and there was an unresolved fail high, I forced the engine to finish the iteration and logged the result to see if it really found a better move.
- if a move failed high ONCE, there was an 88% probability that this will be the new best move (this surprised me - I expected a higher number)
- if a move failed high TWICE, the probability increased to 97.5%
- on THIRD fail high the probability was 99.97%
I found out that if the time is really running out, it is quite safe to accept a new move after it failed high 2x. Obviously, if there is enough time, the safest way is to finish the iteration.
A somewhat related topic was also discussed in this thread
In the aspiration window framework the engine finds a new, better move (e3h6) but the time is not enough to establish the score and the best move is not updated. It plays finally the old best move.
It is possible this a major bug - but I doubt it will have much impact on the strength. Still, I should probably check it out sooner rather than later.
The reason I don't expect it to be major is that once an iteration has started, Komodo is going to try really hard to complete it. So when I fix it I will also do the UCI fail hi and fail low reported the proper way.
I have to wonder how many games Komodo failed to win or draw due to this bug.
I did some research with Critter about handling fail highs at root. I measured the probability of unresolved fail highs being the new best move.
Just to be clear, did you take statistics only when the move was a NEW move and not the same move from the previous iteration?
Whenever the time management decided to stop the search and there was an unresolved fail high, I forced the engine to finish the iteration and logged the result to see if it really found a better move.
- if a move failed high ONCE, there was an 88% probability that this will be the new best move (this surprised me - I expected a higher number)
- if a move failed high TWICE, the probability increased to 97.5%
- on THIRD fail high the probability was 99.97%
I found out that if the time is really running out, it is quite safe to accept a new move after it failed high 2x. Obviously, if there is enough time, the safest way is to finish the iteration.
Capital punishment would be more effective as a preventive measure if it were administered prior to the crime.
rvida wrote:I did some research with Critter about handling fail highs at root. I measured the probability of unresolved fail highs being the new best move. Whenever the time management decided to stop the search and there was an unresolved fail high, I forced the engine to finish the iteration and logged the result to see if it really found a better move.
- if a move failed high ONCE, there was an 88% probability that this will be the new best move (this surprised me - I expected a higher number)
- if a move failed high TWICE, the probability increased to 97.5%
- on THIRD fail high the probability was 99.97%
I found out that if the time is really running out, it is quite safe to accept a new move after it failed high 2x. Obviously, if there is enough time, the safest way is to finish the iteration.
These numbers confirm my own testing. And when you fail-high at the first move there hardly is a risk at all, exceptions are very rare.
What's also interesting to research (changing the topic a bit) are the false fail-high's, often they will appear as best move one or two iterations later after all. In the 90's I tested 2 options:
1. Accept any false fail-high as best move;
2. Research a false fail-high again but now one ply deeper.
Due to the limited hardware of these days plus the fact false fail-high's are rare it was impossible to make an educated decision.
)
