How to go about chasing a bug like this?

[op12no2]

I was watching Lozza play a game this evening, saw a jump in eval and decided to investigate. Usually I assume it's just a horizon thing because Lozza is a relatively slow Javascript engine being interpreted in real-time (i.e. the other engine usually looks deeper).

Pre move:-

[d] 8/4k3/1r6/p5RP/1b1pp3/3p2P1/5P2/1R4K1 w - - 0 61

Post move:-

http://op12no2.me/stuff/l1.jpg

http://op12no2.me/stuff/l2.jpg

As you can see, after Lozza played Rg7+ it all went to pot and for good reason.

If I send exactly the same UCI command (highlighted above from the Arena debug log) to Lozza in isolation (i.e. not in a game) - it finds Rd5 (same as Stockfish).



So how the hell do you track things like that down. I'm guessing it's probably a TT bug. All I can think of is add a whole wodge of assert()s and try and catch inconsistency. I cannot think how to replicate it (*).

The TT is very simple: always replace.

If anybody is interested the source is here: http://op12no2.me/toys/lozzadev/lozza.js

It's a very simple engine and the TT stuff is in ttPut() and ttGet().

Any insights/advice gratefully received... But good news really, a proper bug!

(*)Ah - send all the go commands up to that move as well...?

[op12no2]

(*)Ah - send all the go commands up to that move as well...?

Gosh - it worked - as in replicated the problem.

Code: Select all

1.18 100p | 2.1Mn | 2090312n | 3.6s | g5g7 | Rg7
rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq -
11/23 (44cp) Rg7 Kd8 Rg4 Bc5 Rd1 Re6 Kg2 a4 f3 e4xf3 Kxf3 Re3 
10/22 (52cp) Rg7 Kf8 Rd7 Rd6 Rxd6 Bxd6 h6 d2 h7 Kg7 
10/18 (34cp) Rd5 Rd6 Re5 Re6 Rd5 Rd6 Re5 Re6 Rd5 Rd6 Re5 Re6 Rd5 Rd6 Re5 Re6 Rd5 Rd6 
9/17 (50cp) Rd5 Rd6 Rxd6 Bxd6 h6 d2 h7 Be5 Kg2 
9/17 (34cp) Re5 Re6 Rd5 Rd6 Rxd6 Bxd6 h6 d2 h7 
8/17 (80cp) Re5 Re6 Rxe6 Kxe6 h6 Bd6 h7 Be5 
7/17 (80cp) Re5 Re6 Rxe6 Kxe6 h6 Bd6 h7 Be5 
6/17 (46cp) Re5 Re6 Rxe6 Kxe6 h6 Kd5 h7 
6/17 (9cp) Kg2 d2 Rg4 Re6 f3 e4xf3 
6/15 (8cp) Rd5 Rd6 Re5 Re6 Rxe6 Kxe6 h6 Kd5 
5/15 (20cp) Rd5 Rd6 Rxd6 Kxd6 h6 
5/15 (7cp) Rg7 Kd8 Kg2 d2 f4 Be7 
4/11 (20cp) Rg7 Kd8 Kg2 Rc6 Rf7 
3/8 (33cp) Rg7 Kd8 Kg2 Rc6 
2/7 (41cp) Rg7 Kd8 Rc1 
2/7 (15cp) Rd5 Rd6 Rxd6 Bxd6 Rd1 Ke6 Rxd3 
2/6 (9cp) Re5 Re6 Rb5 Kd6 Rxb4 
1/6 (16cp) Re5 Re6 Rxb4 a5xb4 Rxe6 Kxe6 
1/1 (-18cp) Rd5 

[AlvaroBegue]

I don't know how much of a bug it is. My engine thinks Rg7+ is best at depths 11 and 12, then prefers Rd5. But even mighty Stockfish likes Rg7+ at depth 8.

[op12no2]

I don't know how much of a bug it is. My engine thinks Rg7+ is best at depths 11 and 12, then prefers Rd5. But even mighty Stockfish likes Rg7+ at depth 8.

Hi Álvaro, OK thanks for the info. I think what worries me is the completely different analysis outputs within and outside the game.

It's something to do with history move ordering as well - which I may dump in fact.

[hgm]

Indeed, dependence on previous moves can be a nuisance whe trying to reproduce problems. My engine CrazyWa is still in the stage where I clear hash table and history before every search (killers are self-clearing), to ensure reproducibility for easy debugging. But it is so stable now that I can consider dropping that, to gain a little bit of extra speed.

What sometimes helps when suspect symptoms do not reproduce when searching from the position where they occurred in a game, is starting from the position the move before, make sure it is searched to the same depth as in the game (by giving ample time, and a depth limit), then switching on the debug output (for which I usually build a command into the engine), enter the opponent move (and start a search if this is not automatic). The previous search might be enough to prepare the TT and history toa state that is close enough to reproduce the bug. Sometimes I go back even two full moves. If that doesn't work, I usually give up. When running from the command line it is easy to copy-paste the required set of commands to order the various searches into the engine.

[op12no2]

Thanks HG. I can reproduce it by sending the last 4 go commands but it only happens about 50% of the time. I can 'fix' it by turning off top level window shrinking - the in/out game analysis become similar and stable...

[op12no2]

The traces below show the analysis for the problem move, after 5 half moves leading up to it have loaded up the TT, each simplified to search to depth 12.

Existing code:-

Code: Select all

12/22 (40cp) Rg7 Kf8 Rd7 Rd6 Rxd6 Bxd6 f4 d2 h6 Kg8 Kf2 e3 
11/22 (52cp) Rg7 Kf8 Rd7 Rd6 Rxd6 Bxd6 f4 Kf7 h6 Kg6 Rb6 
10/12 (57cp) Rg7 Kd6 h6 Rb8 Rg6 Kd7 h7 Rh8 Rb6 Rc8 Rb7 
9/10 (57cp) Rg7 Kd6 h6 Rb8 Rg6 Kd7 h7 Rh8 Rb6 Rc8 
8/8 (57cp) Rg7 Kd6 h6 Rb8 Rg6 Kd7 h7 Rh8 
7/8 (57cp) Rg7 Kd6 h6 Rb8 Rg6 Kd7 h7 Rh8 
6/8 (57cp) Rg7 Kd6 h6 Rb8 Rg6 Kd7 h7 Rh8 
5/8 (57cp) Rg7 Kd6 h6 Rb8 Rg6 Kd7 h7 Rh8 
4/6 (57cp) Rg7 Kd6 h6 Rb8 Rg6 Kd7 
3/6 (57cp) Rg7 Kd6 h6 Rb8 Rg6 Kd7 
2/2 (57cp) Rg7 Kd6 
1/1 (57cp) Rg7 

Without top level window shrinking:-

Code: Select all

12/24 (71cp) Rd5 Rd6 Rxd6 Bxd6 h6 d2 h7 Be5 Kg2 a4 f4 Bf6 
11/24 (39cp) Rd5 Rd6 Rxd6 Bxd6 h6 d2 h7 Be5 Kg2 e3 Kf3 
11/24 (33cp) Rg7 Ke8 Rc1 Bc3 Rg4 Re6 h6 d2 Rb1 a4 h7 
10/19 (36cp) Rg7 Kd8 Rg4 Bc5 Rg8 Kd7 Rg7 Kd6 Rg6 Kc7 Rxb6 Bxb6 
9/19 (36cp) Rg7 Kd8 Rg4 Bc5 Rg8 Kd7 Rg7 Kd6 Rg6 Kc7 Rxb6 Bxb6 
8/17 (57cp) Rg7 Ke8 Rg4 Re6 h6 Re7 Rg8 Kd7 Rc1 
7/16 (40cp) Rg7 Kd6 h6 d2 h7 Rb8 Rb7 Rc8 
6/15 (23cp) Rg7 Kd8 Rg4 Re6 Rg7 Rc6 
5/15 (23cp) Rg7 Kd8 Rg4 Re6 Rg7 Rc6 
4/12 (20cp) Rg7 Kd6 Rg6 Kc5 Rc1 Kb5 Rxb6 Kxb6 h6 d2 
3/10 (22cp) Rg7 Kd8 Rc1 Be7 
3/8 (16cp) Rd5 Rd6 Re5 Re6 Rb5 d2 
2/7 (15cp) Rd5 Rd6 Rxd6 Bxd6 Rb7 Kf6 Rf7 
2/6 (9cp) Re5 Re6 Rb5 d2 
1/6 (16cp) Re5 Re6 Rxe6 Kxe6 Rxb4 a5xb4 
1/4 (-19cp) Kg2 d2 Rxb4 d1=Q 
1/1 (-40cp) Rg4 

The latter is also 3 times as fast!

(12/24 means 12 ply search and max ply reached including Q was 24 from root.)