Stockfish bug

Discussion of chess software programming and technical issues.

Moderators: hgm, Rebel, chrisw

Sven
Posts: 4052
Joined: Thu May 15, 2008 9:57 pm
Location: Berlin, Germany
Full name: Sven Schüle

Re: Stockfish bug

Post by Sven »

tpetzke wrote:Yes, I agree. I gave both positions to my engine and iCE scored them also as draws.

The underlying assumption is that if my opponent plays in a position 34. Qf8 once (which enables a DRAW by repetition) he will do so also a 2nd time because he obviously does not realize this will become a draw.

For engines this assumption works pretty well. Humans are less predictable.

Thomas...
Why are you reducing the problem to the level of "assumptions"? The problem I described above is a real game-playing problem, and it can also occur when playing another engine that has the same "always score first repetition as draw" behaviour. I think that "assumptions" do not always help in being precise enough.


Regarding the fix I described above, I forgot to mention another detail that is required for the fix to work. The condition to stop the repetition detection loop is slightly more complex. Actually there should be two loops:

a) a first loop starting at "current ply - 4" and going backwards in steps of 2 plies up to the root node of the current search (but not beyond), stopping at the first repetition (pos.hashKey == currentPos.hashKey) and always scoring that as a draw,

b) a second loop that is only started if the first loop did not encounter a repetition, starting 2 plies backwards from the node that was last visited by the first loop, going backwards in steps of 2 plies up to the beginning of the game, stopping at the first repetition (pos.hashKey == currentPos.hashKey), and only scoring that as a draw if it is the second repetition.

EDIT: of course both loops can stop when crossing the boundary given by the last irreversible move.

For the last part of b) it would be necessary to store the information whether game positions were already repeated, e.g. as a "numberOfRepetitions" once per position (or once per game position if these are kept separately, depending on individual data structures). With this information the second loop would score a repetition as draw if the repeated position (which is a game position above the search root) has "numberOfRepetitions > 0" so that we now have at least the second repetition.

Of course it is also possible, and may be even simpler, not to store the number of repetitions but instead to continue the second loop after the first repetition and to only stop at the second one.

Also it may be possible that there is some clever solution to combine both loops in one, although I would not recommend that.

Does that sound reasonable? I think we already discussed that detail a couple of months ago but I don't recall whether we found this type of solution back then.

Sven
Last edited by Sven on Tue Jun 11, 2013 12:13 pm, edited 1 time in total.
Michel
Posts: 2272
Joined: Mon Sep 29, 2008 1:50 am

Re: Stockfish bug

Post by Michel »

For a patch to be applied to stockfish it has to either

(1) produce elo gain (and as a secondary criterion: be sufficiently simple);
(2) simplify code and not change node count.

One may debate if these criteria are optimal, but since Stockfish is the strongest
open source engine around, they must count for something.

A patch that fixes the issue under discussion was tested on fishtest. But since neither
(1) or (2) applies to it, it was not applied.
Sven
Posts: 4052
Joined: Thu May 15, 2008 9:57 pm
Location: Berlin, Germany
Full name: Sven Schüle

Re: Stockfish bug

Post by Sven »

Uri Blass wrote:
stevenaaus wrote:No, it's definitely a bug. There is no draw by repetition.
This is stopping me from using Stockfish as my preferred engine, as I'm hitting this bug everyday.
No
It is not a bug because stockfish is not designed for you.

A bug is something that does not perform as the programmer expect.
Not something that does not perform as you expect.

Programmer of stockfish care about playing strength and not about analysis
and if you have no way to fix it that improve playing strength it is going to behave in the same way in the future.
In the light of all that I wrote today in this thread (in parallel subthreads), can you explain why you think that the given case can never hurt playing strength? I strongly believe that evaluating a move as draw that loses in fact by being mated in 9 plies can definitely affect playing strength, so this is NOT an "analysis only" issue.

Sven
Sven
Posts: 4052
Joined: Thu May 15, 2008 9:57 pm
Location: Berlin, Germany
Full name: Sven Schüle

Re: Stockfish bug

Post by Sven »

Michel wrote:For a patch to be applied to stockfish it has to either

(1) produce elo gain (and as a secondary criterion: be sufficiently simple);
(2) simplify code and not change node count.

One may debate if these criteria are optimal, but since Stockfish is the strongest
open source engine around, they must count for something.

A patch that fixes the issue under discussion was tested on fishtest. But since neither
(1) or (2) applies to it, it was not applied.
I do not understand (2). Any bugfix in search will potentially change the node count.

I also do not understand why (1) should not be extended to:
"produce elo gain, or fix an obvious bug without losing elo".

Sven
Sven
Posts: 4052
Joined: Thu May 15, 2008 9:57 pm
Location: Berlin, Germany
Full name: Sven Schüle

Re: Stockfish bug

Post by Sven »

Sven Schüle wrote:b) a second loop that is only started if the first loop did not encounter a repetition, starting 2 plies backwards from the node that was last visited by the first loop, going backwards in steps of 2 plies up to the beginning of the game, stopping at the first repetition (pos.hashKey == currentPos.hashKey), and only scoring that as a draw if it is the second repetition.
Better wording (no edit possible any longer):
... stopping at the first repetition encountered (pos.hashKey == currentPos.hashKey), and only scoring that as a draw if it is the second repetition in total.
Michel
Posts: 2272
Joined: Mon Sep 29, 2008 1:50 am

Re: Stockfish bug

Post by Michel »

I do not understand (2). Any bugfix in search will potentially change the node count.
produce elo gain, or fix an obvious bug without losing elo"

The contradiction is resolved if you define a bug as something that causes elo loss...

It is a semantic issue. The problem under discussion is not viewed as a bug by the Stockfish developers since fixing it does not produce elo gain....
stevenaaus
Posts: 608
Joined: Wed Oct 13, 2010 9:44 am
Location: Australia

Re: Stockfish bug

Post by stevenaaus »

Uri Blass wrote:
stevenaaus wrote:No, it's definitely a bug. There is no draw by repetition.
This is stopping me from using Stockfish as my preferred engine, as I'm hitting this bug everyday.
No
It is not a bug because stockfish is not designed for you.

A bug is something that does not perform as the programmer expect.
Not something that does not perform as you expect.
:) Spoken like a true programmer.
Great software however is written with the user in mind.

But of course I understand if they care to concentrate on Computer Tournaments.
tpetzke
Posts: 686
Joined: Thu Mar 03, 2011 4:57 pm
Location: Germany

Re: Stockfish bug

Post by tpetzke »

Hi Sven,

it does not hurt much because in an engine - engine match this position will likely not occur.

Either the white engine "sees" that its move 44. Qf8 will enable a draw by repetition and will play 44. Qd5 instead. Or it does not see it (unlikely) and will play Qf8 again towards the draw.

That an engine chooses different moves in the same position is unlikely. It is assumed it chooses the same. We can't be sure, because on the 2nd occurrence the engine might be able to search deeper and produce a different move. But in most cases it will make the same.

Also the transposition table is blind to repetition problems and we could solve that by storing path information. But it will not make the program stronger so I guess nobody does that.

Do you ?

Thomas...
User avatar
Evert
Posts: 2929
Joined: Sat Jan 22, 2011 12:42 am
Location: NL

Re: Stockfish bug

Post by Evert »

tpetzke wrote: That an engine chooses different moves in the same position is unlikely. It is assumed it chooses the same. We can't be sure, because on the 2nd occurrence the engine might be able to search deeper and produce a different move. But in most cases it will make the same.
I wonder if this could be exploited: fool the other engine into thinking that it's going to get a repetition, but instead it ends up in an inferior position where the first engine avoids the repetition the second time.

Probably hard to put into a working algorithm that actually gains elo...
User avatar
hgm
Posts: 27808
Joined: Fri Mar 10, 2006 10:06 am
Location: Amsterdam
Full name: H G Muller

Re: Stockfish bug

Post by hgm »

Sven Schüle wrote:Also it may be possible that there is some clever solution to combine both loops in one, although I would not recommend that.
In Joker I hash the last stretch of reversible moves into a 256-entry table. This table stores the key and the repeat count. (Count = 0 indicates the entry is empty.) This way you find the position always with a single loop, which either finds the current position, or an empty entry (usually on the first try).

With this system you can score a draw when you hit a position that has count >= 2. If the position has count < 2 you would increment that count by 2 in search, and by 1 at game level. Leaving the position you decrement its count again by 2 to restore the original value (as this only happens in search).