I was watching Lozza play a game this evening, saw a jump in eval and decided to investigate. Usually I assume it's just a horizon thing because Lozza is a relatively slow Javascript engine being interpreted in real-time (i.e. the other engine usually looks deeper).
Pre move:-
[d] 8/4k3/1r6/p5RP/1b1pp3/3p2P1/5P2/1R4K1 w - - 0 61
Post move:-
http://op12no2.me/stuff/l1.jpg
http://op12no2.me/stuff/l2.jpg
As you can see, after Lozza played Rg7+ it all went to pot and for good reason.
If I send exactly the same UCI command (highlighted above from the Arena debug log) to Lozza in isolation (i.e. not in a game) - it finds Rd5 (same as Stockfish).
So how the hell do you track things like that down. I'm guessing it's probably a TT bug. All I can think of is add a whole wodge of assert()s and try and catch inconsistency. I cannot think how to replicate it (*).
The TT is very simple: always replace.
If anybody is interested the source is here: http://op12no2.me/toys/lozzadev/lozza.js
It's a very simple engine and the TT stuff is in ttPut() and ttGet().
Any insights/advice gratefully received... But good news really, a proper bug!
(*)Ah - send all the go commands up to that move as well...?
How to go about chasing a bug like this?
Moderators: hgm, Rebel, chrisw
-
- Posts: 490
- Joined: Tue Feb 04, 2014 12:25 pm
- Full name: Colin Jenkins
-
- Posts: 490
- Joined: Tue Feb 04, 2014 12:25 pm
- Full name: Colin Jenkins
Re: How to go about chasing a bug like this?
Gosh - it worked - as in replicated the problem.op12no2 wrote: (*)Ah - send all the go commands up to that move as well...?
Code: Select all
1.18 100p | 2.1Mn | 2090312n | 3.6s | g5g7 | Rg7
rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq -
11/23 (44cp) Rg7 Kd8 Rg4 Bc5 Rd1 Re6 Kg2 a4 f3 e4xf3 Kxf3 Re3
10/22 (52cp) Rg7 Kf8 Rd7 Rd6 Rxd6 Bxd6 h6 d2 h7 Kg7
10/18 (34cp) Rd5 Rd6 Re5 Re6 Rd5 Rd6 Re5 Re6 Rd5 Rd6 Re5 Re6 Rd5 Rd6 Re5 Re6 Rd5 Rd6
9/17 (50cp) Rd5 Rd6 Rxd6 Bxd6 h6 d2 h7 Be5 Kg2
9/17 (34cp) Re5 Re6 Rd5 Rd6 Rxd6 Bxd6 h6 d2 h7
8/17 (80cp) Re5 Re6 Rxe6 Kxe6 h6 Bd6 h7 Be5
7/17 (80cp) Re5 Re6 Rxe6 Kxe6 h6 Bd6 h7 Be5
6/17 (46cp) Re5 Re6 Rxe6 Kxe6 h6 Kd5 h7
6/17 (9cp) Kg2 d2 Rg4 Re6 f3 e4xf3
6/15 (8cp) Rd5 Rd6 Re5 Re6 Rxe6 Kxe6 h6 Kd5
5/15 (20cp) Rd5 Rd6 Rxd6 Kxd6 h6
5/15 (7cp) Rg7 Kd8 Kg2 d2 f4 Be7
4/11 (20cp) Rg7 Kd8 Kg2 Rc6 Rf7
3/8 (33cp) Rg7 Kd8 Kg2 Rc6
2/7 (41cp) Rg7 Kd8 Rc1
2/7 (15cp) Rd5 Rd6 Rxd6 Bxd6 Rd1 Ke6 Rxd3
2/6 (9cp) Re5 Re6 Rb5 Kd6 Rxb4
1/6 (16cp) Re5 Re6 Rxb4 a5xb4 Rxe6 Kxe6
1/1 (-18cp) Rd5
-
- Posts: 931
- Joined: Tue Mar 09, 2010 3:46 pm
- Location: New York
- Full name: Álvaro Begué (RuyDos)
Re: How to go about chasing a bug like this?
I don't know how much of a bug it is. My engine thinks Rg7+ is best at depths 11 and 12, then prefers Rd5. But even mighty Stockfish likes Rg7+ at depth 8.
-
- Posts: 490
- Joined: Tue Feb 04, 2014 12:25 pm
- Full name: Colin Jenkins
Re: How to go about chasing a bug like this?
Hi Álvaro, OK thanks for the info. I think what worries me is the completely different analysis outputs within and outside the game.AlvaroBegue wrote:I don't know how much of a bug it is. My engine thinks Rg7+ is best at depths 11 and 12, then prefers Rd5. But even mighty Stockfish likes Rg7+ at depth 8.
It's something to do with history move ordering as well - which I may dump in fact.
-
- Posts: 27808
- Joined: Fri Mar 10, 2006 10:06 am
- Location: Amsterdam
- Full name: H G Muller
Re: How to go about chasing a bug like this?
Indeed, dependence on previous moves can be a nuisance whe trying to reproduce problems. My engine CrazyWa is still in the stage where I clear hash table and history before every search (killers are self-clearing), to ensure reproducibility for easy debugging. But it is so stable now that I can consider dropping that, to gain a little bit of extra speed.
What sometimes helps when suspect symptoms do not reproduce when searching from the position where they occurred in a game, is starting from the position the move before, make sure it is searched to the same depth as in the game (by giving ample time, and a depth limit), then switching on the debug output (for which I usually build a command into the engine), enter the opponent move (and start a search if this is not automatic). The previous search might be enough to prepare the TT and history toa state that is close enough to reproduce the bug. Sometimes I go back even two full moves. If that doesn't work, I usually give up. When running from the command line it is easy to copy-paste the required set of commands to order the various searches into the engine.
What sometimes helps when suspect symptoms do not reproduce when searching from the position where they occurred in a game, is starting from the position the move before, make sure it is searched to the same depth as in the game (by giving ample time, and a depth limit), then switching on the debug output (for which I usually build a command into the engine), enter the opponent move (and start a search if this is not automatic). The previous search might be enough to prepare the TT and history toa state that is close enough to reproduce the bug. Sometimes I go back even two full moves. If that doesn't work, I usually give up. When running from the command line it is easy to copy-paste the required set of commands to order the various searches into the engine.
-
- Posts: 490
- Joined: Tue Feb 04, 2014 12:25 pm
- Full name: Colin Jenkins
Re: How to go about chasing a bug like this?
Thanks HG. I can reproduce it by sending the last 4 go commands but it only happens about 50% of the time. I can 'fix' it by turning off top level window shrinking - the in/out game analysis become similar and stable...
-
- Posts: 490
- Joined: Tue Feb 04, 2014 12:25 pm
- Full name: Colin Jenkins
Re: How to go about chasing a bug like this?
The traces below show the analysis for the problem move, after 5 half moves leading up to it have loaded up the TT, each simplified to search to depth 12.
Existing code:-
Without top level window shrinking:-
The latter is also 3 times as fast!
(12/24 means 12 ply search and max ply reached including Q was 24 from root.)
Existing code:-
Code: Select all
12/22 (40cp) Rg7 Kf8 Rd7 Rd6 Rxd6 Bxd6 f4 d2 h6 Kg8 Kf2 e3
11/22 (52cp) Rg7 Kf8 Rd7 Rd6 Rxd6 Bxd6 f4 Kf7 h6 Kg6 Rb6
10/12 (57cp) Rg7 Kd6 h6 Rb8 Rg6 Kd7 h7 Rh8 Rb6 Rc8 Rb7
9/10 (57cp) Rg7 Kd6 h6 Rb8 Rg6 Kd7 h7 Rh8 Rb6 Rc8
8/8 (57cp) Rg7 Kd6 h6 Rb8 Rg6 Kd7 h7 Rh8
7/8 (57cp) Rg7 Kd6 h6 Rb8 Rg6 Kd7 h7 Rh8
6/8 (57cp) Rg7 Kd6 h6 Rb8 Rg6 Kd7 h7 Rh8
5/8 (57cp) Rg7 Kd6 h6 Rb8 Rg6 Kd7 h7 Rh8
4/6 (57cp) Rg7 Kd6 h6 Rb8 Rg6 Kd7
3/6 (57cp) Rg7 Kd6 h6 Rb8 Rg6 Kd7
2/2 (57cp) Rg7 Kd6
1/1 (57cp) Rg7
Code: Select all
12/24 (71cp) Rd5 Rd6 Rxd6 Bxd6 h6 d2 h7 Be5 Kg2 a4 f4 Bf6
11/24 (39cp) Rd5 Rd6 Rxd6 Bxd6 h6 d2 h7 Be5 Kg2 e3 Kf3
11/24 (33cp) Rg7 Ke8 Rc1 Bc3 Rg4 Re6 h6 d2 Rb1 a4 h7
10/19 (36cp) Rg7 Kd8 Rg4 Bc5 Rg8 Kd7 Rg7 Kd6 Rg6 Kc7 Rxb6 Bxb6
9/19 (36cp) Rg7 Kd8 Rg4 Bc5 Rg8 Kd7 Rg7 Kd6 Rg6 Kc7 Rxb6 Bxb6
8/17 (57cp) Rg7 Ke8 Rg4 Re6 h6 Re7 Rg8 Kd7 Rc1
7/16 (40cp) Rg7 Kd6 h6 d2 h7 Rb8 Rb7 Rc8
6/15 (23cp) Rg7 Kd8 Rg4 Re6 Rg7 Rc6
5/15 (23cp) Rg7 Kd8 Rg4 Re6 Rg7 Rc6
4/12 (20cp) Rg7 Kd6 Rg6 Kc5 Rc1 Kb5 Rxb6 Kxb6 h6 d2
3/10 (22cp) Rg7 Kd8 Rc1 Be7
3/8 (16cp) Rd5 Rd6 Re5 Re6 Rb5 d2
2/7 (15cp) Rd5 Rd6 Rxd6 Bxd6 Rb7 Kf6 Rf7
2/6 (9cp) Re5 Re6 Rb5 d2
1/6 (16cp) Re5 Re6 Rxe6 Kxe6 Rxb4 a5xb4
1/4 (-19cp) Kg2 d2 Rxb4 d1=Q
1/1 (-40cp) Rg4
(12/24 means 12 ply search and max ply reached including Q was 24 from root.)