Transposition table bug - Points to incorrect alpha-beta?

Discussion of chess software programming and technical issues.

Moderators: hgm, Rebel, chrisw

bob
Posts: 20943
Joined: Mon Feb 27, 2006 7:30 pm
Location: Birmingham, AL

Re: Transposition table bug - Points to incorrect alpha-beta

Post by bob »

rob wrote:I put a copy of my negamax ab + TT implementation online, with just a bit of simplification, removal of some unnecessary details (like how draw by 3-fold repetition is detected).

It's essentially a clone of Tony Marsland's recipe given in Figure 7 of his paper "A Review of Game Tree Pruning".

negamax_ab_TT

Rob
One thing that certainly looks wrong, for starters, is your mate score. Mates are measured in plies from the root, not from the tips. Most would do something like this:

if (checkmate)
return INFINITY - ply;

which turns into a value that represents the distance to mate from the root. Depth is usually "distance from here to the tip (assuming there are no extensions/reductions applied)", which really doesn't belong in the mate score computation, unless I misread your use of depth.
Sven
Posts: 4052
Joined: Thu May 15, 2008 9:57 pm
Location: Berlin, Germany
Full name: Sven Schüle

Re: Transposition table bug - Points to incorrect alpha-beta

Post by Sven »

bob wrote:
rob wrote:I put a copy of my negamax ab + TT implementation online, with just a bit of simplification, removal of some unnecessary details (like how draw by 3-fold repetition is detected).

It's essentially a clone of Tony Marsland's recipe given in Figure 7 of his paper "A Review of Game Tree Pruning".

negamax_ab_TT

Rob
One thing that certainly looks wrong, for starters, is your mate score. Mates are measured in plies from the root, not from the tips. Most would do something like this:

if (checkmate)
return INFINITY - ply;

which turns into a value that represents the distance to mate from the root. Depth is usually "distance from here to the tip (assuming there are no extensions/reductions applied)", which really doesn't belong in the mate score computation, unless I misread your use of depth.
-(INFINITY - (max_depth - depthleft))
would intrinsically be the same as
-(INFINITY - distance_from_root)
if either no extensions/reductions are used, or max_depth grows and shrinks with each extension or reduction. I agree that using distance_from_root (or "ply") is the much better solution on the long term (requiring another parameter of the search function which is trivial of course). But if we assume that no extensions/reductions are used in the current testing stage of the OP then this is not yet the explanation of his bug but "only" a valid improvement.
Sven
Posts: 4052
Joined: Thu May 15, 2008 9:57 pm
Location: Berlin, Germany
Full name: Sven Schüle

Re: Transposition table bug - Points to incorrect alpha-beta

Post by Sven »

rob wrote:I put a copy of my negamax ab + TT implementation online, with just a bit of simplification, removal of some unnecessary details (like how draw by 3-fold repetition is detected).

It's essentially a clone of Tony Marsland's recipe given in Figure 7 of his paper "A Review of Game Tree Pruning".

negamax_ab_TT

Rob
In case of a beta cutoff with score == beta you leave the flag_type as "exact" (VALID) due to the condition "if (score > beta) ...", I'd assume it should be set to LBOUND?
bob
Posts: 20943
Joined: Mon Feb 27, 2006 7:30 pm
Location: Birmingham, AL

Re: Transposition table bug - Points to incorrect alpha-beta

Post by bob »

Sven Schüle wrote:
bob wrote:
rob wrote:I put a copy of my negamax ab + TT implementation online, with just a bit of simplification, removal of some unnecessary details (like how draw by 3-fold repetition is detected).

It's essentially a clone of Tony Marsland's recipe given in Figure 7 of his paper "A Review of Game Tree Pruning".

negamax_ab_TT

Rob
One thing that certainly looks wrong, for starters, is your mate score. Mates are measured in plies from the root, not from the tips. Most would do something like this:

if (checkmate)
return INFINITY - ply;

which turns into a value that represents the distance to mate from the root. Depth is usually "distance from here to the tip (assuming there are no extensions/reductions applied)", which really doesn't belong in the mate score computation, unless I misread your use of depth.
-(INFINITY - (max_depth - depthleft))
would intrinsically be the same as
-(INFINITY - distance_from_root)
if either no extensions/reductions are used, or max_depth grows and shrinks with each extension or reduction. I agree that using distance_from_root (or "ply") is the much better solution on the long term (requiring another parameter of the search function which is trivial of course). But if we assume that no extensions/reductions are used in the current testing stage of the OP then this is not yet the explanation of his bug but "only" a valid improvement.
Think about that again. depth-left is continually changed. It can INCREASE due to a check. It can be reduced by LMR. That's not going to give you distance from the root any significant percentage of the time. "depth" is the wrong value. Distance from the root is a value that changes by exactly one for each move played beyond the root. Depth-remaining has no such relationship except when a program does zero reductions and zero extensions, not to mention zero null-move searches and such...

If he is not extending or reducing anywhere, he can get away with this, but only until he adds his first extension, where he starts to debug inconsistent mate scores once again.

Some of what I saw I didn't understand, such as "don't update bounds or score (I don't remember the exact comment now) in selective search." No idea what that is about.

I really wasn't proposing the above as a reason for his inverted score bug. But I wanted to understand his code and what he thought it was doing as I worked through it. I've always been of the mindset that I am not going to debug through a clear problem and continue, because that problem certainly results in an issue and I don't really want to debug the interaction between different bugs, but rather want to eliminate every bug I see, when I encounter it.

My first test were this my program would be to get a known working program and compare results for known forced mates. In Crafty, one could disable extensions and reductions by simple command-line arguments, to see what it does with a given forced mate. Until I matched Crafty with these positions, I would not add anything new...
Sven
Posts: 4052
Joined: Thu May 15, 2008 9:57 pm
Location: Berlin, Germany
Full name: Sven Schüle

Re: Transposition table bug - Points to incorrect alpha-beta

Post by Sven »

bob wrote:
Sven Schüle wrote:
bob wrote:
rob wrote:I put a copy of my negamax ab + TT implementation online, with just a bit of simplification, removal of some unnecessary details (like how draw by 3-fold repetition is detected).

It's essentially a clone of Tony Marsland's recipe given in Figure 7 of his paper "A Review of Game Tree Pruning".

negamax_ab_TT

Rob
One thing that certainly looks wrong, for starters, is your mate score. Mates are measured in plies from the root, not from the tips. Most would do something like this:

if (checkmate)
return INFINITY - ply;

which turns into a value that represents the distance to mate from the root. Depth is usually "distance from here to the tip (assuming there are no extensions/reductions applied)", which really doesn't belong in the mate score computation, unless I misread your use of depth.
-(INFINITY - (max_depth - depthleft))
would intrinsically be the same as
-(INFINITY - distance_from_root)
if either no extensions/reductions are used, or max_depth grows and shrinks with each extension or reduction. I agree that using distance_from_root (or "ply") is the much better solution on the long term (requiring another parameter of the search function which is trivial of course). But if we assume that no extensions/reductions are used in the current testing stage of the OP then this is not yet the explanation of his bug but "only" a valid improvement.
Think about that again. depth-left is continually changed. It can INCREASE due to a check. It can be reduced by LMR. That's not going to give you distance from the root any significant percentage of the time. "depth" is the wrong value. Distance from the root is a value that changes by exactly one for each move played beyond the root. Depth-remaining has no such relationship except when a program does zero reductions and zero extensions, not to mention zero null-move searches and such...

If he is not extending or reducing anywhere, he can get away with this, but only until he adds his first extension, where he starts to debug inconsistent mate scores once again.

Some of what I saw I didn't understand, such as "don't update bounds or score (I don't remember the exact comment now) in selective search." No idea what that is about.

I really wasn't proposing the above as a reason for his inverted score bug. But I wanted to understand his code and what he thought it was doing as I worked through it. I've always been of the mindset that I am not going to debug through a clear problem and continue, because that problem certainly results in an issue and I don't really want to debug the interaction between different bugs, but rather want to eliminate every bug I see, when I encounter it.

My first test were this my program would be to get a known working program and compare results for known forced mates. In Crafty, one could disable extensions and reductions by simple command-line arguments, to see what it does with a given forced mate. Until I matched Crafty with these positions, I would not add anything new...
Good to know that we both agree upon all that you wrote on this topic, as well as what I wrote :-) It is definitely a necessary fix for the OP's program to use "ply" instead of his current implementation, although that will most probably not fix his current bug yet.

Only one exception where I disagree to you: In my mindset it can sometimes be better *not* to apply other corrections of which I believe they are necessary but do not help to fix the original bug, since doing so can change the behaviour of my program in a way that makes finding the bug even harder. It *can* be better not to do it in my experience, and it *can* be better to do it in your experience. So it depends. When in doubt, I try to change as few as possible until I actually catch the bug.

Sven
bob
Posts: 20943
Joined: Mon Feb 27, 2006 7:30 pm
Location: Birmingham, AL

Re: Transposition table bug - Points to incorrect alpha-beta

Post by bob »

My take is that this is a VERY bad practice. I don't want to debug a program when it already has a bunch of known bugs, in addition to the bug I am actually looking for. Bugs are, by definition, pieces of code that either don't do what they are supposed to do, or do what they are supposed to do but also do something "extra" and unwanted.

How do you know that the bug you just spotted but chose to ignore is NOT affecting (or even partially causing) the bug you are actually searching for? For example, you hear a strange noise coming from your car. When you start to investigate, you find the oil pressure is low. Do you ignore that? It might be the sending unit. It might be the oil level, where low oil can and will cause unusual sounds. It might be the oil pump where low oil pressure will certainly cause unusual sounds...

I'd jump right on that first, and then move on if the noise remains after the oil problem is fixed.
User avatar
hgm
Posts: 27790
Joined: Fri Mar 10, 2006 10:06 am
Location: Amsterdam
Full name: H G Muller

Re: Transposition table bug - Points to incorrect alpha-beta

Post by hgm »

A reproducible error is worth its weight in gold. I would fix the source of that first, because I know I can always do it.
bob
Posts: 20943
Joined: Mon Feb 27, 2006 7:30 pm
Location: Birmingham, AL

Re: Transposition table bug - Points to incorrect alpha-beta

Post by bob »

hgm wrote:A reproducible error is worth its weight in gold. I would fix the source of that first, because I know I can always do it.
Different strokes for different folks. I'm not sure how you could be sure the bug you just spotted is not part of the problem, unless you have carefully looked at this "unexpected bug" to be certain it is not. And if you had looked at it that carefully, why not fix it while it is on your mind, rather than taking a chance on forgetting about it?
Sven
Posts: 4052
Joined: Thu May 15, 2008 9:57 pm
Location: Berlin, Germany
Full name: Sven Schüle

Re: Transposition table bug - Points to incorrect alpha-beta

Post by Sven »

bob wrote:My take is that this is a VERY bad practice. I don't want to debug a program when it already has a bunch of known bugs, in addition to the bug I am actually looking for. Bugs are, by definition, pieces of code that either don't do what they are supposed to do, or do what they are supposed to do but also do something "extra" and unwanted.

How do you know that the bug you just spotted but chose to ignore is NOT affecting (or even partially causing) the bug you are actually searching for? For example, you hear a strange noise coming from your car. When you start to investigate, you find the oil pressure is low. Do you ignore that? It might be the sending unit. It might be the oil level, where low oil can and will cause unusual sounds. It might be the oil pump where low oil pressure will certainly cause unusual sounds...

I'd jump right on that first, and then move on if the noise remains after the oil problem is fixed.
May be fine for cars. Not necessarily for software, since software is different: you can change everything, undo the change, have different versions, and compare them to see how they behave. In my experience with debugging of software you often end up making lots of "really necessary" or "nice to have" changes while the original problem remains there, and once you finally catch it you now start to undo all those other changes to prove (or let's say, to check) whether they were actually unrelated to the bug or not. It is quite uncomfortable to state "this change together with all those other changes fixes the bug but I don't know exactly which one is the important fix that is sufficient to let the problem go away" because it does not help to understand the source of the bug.
bob
Posts: 20943
Joined: Mon Feb 27, 2006 7:30 pm
Location: Birmingham, AL

Re: Transposition table bug - Points to incorrect alpha-beta

Post by bob »

Sven Schüle wrote:
bob wrote:My take is that this is a VERY bad practice. I don't want to debug a program when it already has a bunch of known bugs, in addition to the bug I am actually looking for. Bugs are, by definition, pieces of code that either don't do what they are supposed to do, or do what they are supposed to do but also do something "extra" and unwanted.

How do you know that the bug you just spotted but chose to ignore is NOT affecting (or even partially causing) the bug you are actually searching for? For example, you hear a strange noise coming from your car. When you start to investigate, you find the oil pressure is low. Do you ignore that? It might be the sending unit. It might be the oil level, where low oil can and will cause unusual sounds. It might be the oil pump where low oil pressure will certainly cause unusual sounds...

I'd jump right on that first, and then move on if the noise remains after the oil problem is fixed.
May be fine for cars. Not necessarily for software, since software is different: you can change everything, undo the change, have different versions, and compare them to see how they behave. In my experience with debugging of software you often end up making lots of "really necessary" or "nice to have" changes while the original problem remains there, and once you finally catch it you now start to undo all those other changes to prove (or let's say, to check) whether they were actually unrelated to the bug or not. It is quite uncomfortable to state "this change together with all those other changes fixes the bug but I don't know exactly which one is the important fix that is sufficient to let the problem go away" because it does not help to understand the source of the bug.
Note I was talking about an observed "bug" and not just "an enhancement." When I spot a bug, whether it is what I am looking for or not, I fix it. If that somehow masks the original bug, I can always remove the change temporarily to expose it again.

I don't do much "new feature" work while trying to find a bug. I'd rather find just one bug than have to find that plus two new ones.