My troubles with MultiPV and Syzygy in Stockfish 7

[Arpad Rusz]

Recently I was analyzing lots of chess studies with Stockfish 7 (or with its Matefinder version) from the latest Endgame Study Database V ( http://hhdbv.nl/). These engines make a very good job, but I have encountered a disturbing problem when analyzing positions with 5 or 6 men. Given some conditions, in the MultiPV output not only the drawing or losing moves but even some of the winning moves could be missing...

Case 1: Cursed Wins

Studies don't respect the 50 move rule, so I leave Syzygy50MoveRule unchecked. That way the engine can report a win. E.g.

[D]2R5/8/8/8/B3n3/3n4/K2k4/8 w - - 0 1

Stockfish reports only one move (Ka1) even in MultiPV mode. Does it means that only Ka1!! wins? It would be nice, but no! According to the online syzygy tablebases, there are 4 winning moves. https://syzygy-tables.info/?fen=2R5/8/8 ... 3n4/K2k4/8 w - - 0 1
Maybe it reports only the best move? Let's see! The following position has 17 winning moves, see https://syzygy-tables.info/?fen=2R5/8/8 ... /3nk3/8/K7 w - - 0 1.

[D]2R5/8/8/8/B3n3/3nk3/8/K7 w - - 0 1

Guess, how many of the are shown by the engine? Well, the winner is: 2! The two best moves Rd8 and Re8, both with equal DTZ, are reported.

Case 2: Repetitions

Many engine report 0.00 if a position is reached the second/third time. Even if sometimes disturbing, that could seem logical. But I have observed a different phenomenon regarding repetitions. If early in the game there was a repetition, after that MultiPV outputs only the best DTZ move/moves in the entire game. Example: starting with position A let us reach position B in two ways.

Position A
[D]4n3/6p1/6R1/1b6/8/8/1K1k4/8 w - - 0 1

Position B
[D]4n3/6p1/4R3/1b6/8/8/1K1k4/8 b - - 0 1

First way: 1.Re6
Second way (after a repetition): 1. Rg5 Bc6 2. Rg6 Bb5 3. Re6

There are only two winning moves in the Position B:
https://syzygy-tables.info/?fen=4n3/6p1 ... /8/1K1k4/8 b - - 0 1

In the first case Stockfish shows both of them (Nf6 and Kd3), but when reaching Position B by the second method, reports only the best move (Nf6).

The two problems seems to be related.

If a user wants to analyze a position seting MultiPV to some value N, that means he/she would like to see N variations, no matter what! (Of course, with the only exception of not having enough legal moves). Unlike Stockfish, Komodo is fine, but that engine has other issues... It is sad that both these great playing engines are not as good as analysis tools.

[syzygy]

[D]2R5/8/8/8/B3n3/3n4/K2k4/8 w - - 0 1

Stockfish reports only one move (Ka1) even in MultiPV mode. Does it means that only Ka1!! wins? It would be nice, but no! According to the online syzygy tablebases, there are 4 winning moves.

That is because the distance to zero is so large that the code decides to consider only the DTZ-optimal moves.

I agree this does not play very nice with multipv, but without this SF would end up running in circles (in gameplay). E.g. Bd7 (with DTZ=416) could easily end up scoring better than Ka1 (with DTZ=408) in multipv and in that case SF would play Bd7.

Same for repetitions: if a position gets repeated, that shows that SF does not know how to make progress: it has been running in a circle. The code therefore switches to considering DTZ-optimal moves only (which guarantees progress).

[hgm]

Isn't the whole idea of multi-PV that you also consider sub-optimal moves? It is not like you would ever want to play such a move.

I don't understand how a DTZ=416 move could score better than a DTZ=408 move. That seems a bug. Chess engines are supposed to select moves by score. Not by assigning meaningless scores and then select on something else to prevent the moves with the best scores can be played. If only DTZ=416 moves should be eligible for playing, a scoring that does not give those a higher score than all the others is simply broken.

[syzygy]

Isn't the whole idea of multi-PV that you also consider sub-optimal moves? It is not like you would ever want to play such a move.

Sure, but without the preselection suboptimal moves may move to the top of the list.

I don't understand how a DTZ=416 move could score better than a DTZ=408 move.

Because they are being searched without probing the TBs.

This is nonsense if DTZ > 400, but it makes a lot of sense in normal 6-piece positions where you don't want to rate pawn moves above a mate in 2.

Of course searching without probing the TBs also is not going to give very sensible analysis for positions with DTZ > 400. Arpad might get more useful information out of the online interface.

[Mike S.]

This problem is familiar to me and it indeed seems being a flaw. If an engine is run in MultiPV(x) mode, it is supposed to deliever x variations sorted by eval, under all circumstances (except the number of legal moves is smaller). E.g. if I set MultiPV(3) but I get only one variation, something is broken.

But this is criticism on a luxury level Thank you mucho for those tables! The rest are minor details.

[hgm]

This is nonsense if DTZ > 400, but it makes a lot of sense in normal 6-piece positions where you don't want to rate pawn moves above a mate in 2.

Of course searching without probing the TBs also is not going to give very sensible analysis for positions with DTZ > 400. Arpad might get more useful information out of the online interface.

It still seems a hack to 'cure' one error with another. If a mate in 2 is preferable over a DTZ=1 move, the proper way to do it is to make sure that the mate-in-2 move gets a higher score. The whole point of this probeless search seems to be to allow the engine to prefer moves that are not DTZ optimal, like mates in 2. This is subverted when you then restrict the search to moves that are DTZ optimal, making the search pointless.

And none of that seems a valid excuse for showing too few PVs. To be compliant it should show the N moves with the best DTZ using DTZ as the primary sort key. And it should be easy to correct the scores in this situation so that this would be their natural order.

I don't see any reason yet why you should pay any attention to the score at all if it is not a mate score. It seems always better to just play by DTZ if no mate could be detected.

[kbhearn]

I don't see any reason yet why you should pay any attention to the score at all if it is not a mate score. It seems always better to just play by DTZ if no mate could be detected.

The point is aesthetic. In won endings playing by dtz alone once the root is a tablebase position can result in all kinds of correct but annoying behavior, like saccing your piece for a pawn to get to another won ending when if you'd taken a couple more moves you could've just won the pawn and kept the piece. Hence the preference to do a normally scored nontb search while restricting it to dtz-acceptable moves at root once in a root tb position.

It does seem like a further hack could be added that after dtz-acceptable moves are ordered it could search 'the rest of the moves' if it needs more multi-pv entries.

[hgm]

Still, the correct way to get this behavior would be to score the tablebase positions such that positions in different end-games by their material advantage or pawn advance (provided they are won positions, of course), the DTZ only determining the ranking within a single EGT. Then it would automatically prefer winning the Pawn over sacrificing a Bishop for it, or just fooling around in the current P-slice to be able to reach a DTZ=1 position in the leaves.

[syzygy]

If a mate in 2 is preferable over a DTZ=1 move, the proper way to do it is to make sure that the mate-in-2 move gets a higher score. The whole point of this probeless search seems to be to allow the engine to prefer moves that are not DTZ optimal, like mates in 2. This is subverted when you then restrict the search to moves that are DTZ optimal, making the search pointless.

The search is not restricted to DTZ-optimal moves, but to moves that preserve the outcome of the game. So in a winning position moves that are not guaranteed to win within the 50-move counter are eliminated.

Now the question is what to do with "cursed win" positions. They are theoretically drawn, so you could just search all non-losing moves. But it seems smarter to hope for inaccurate play by the opponent that perhaps will allow you to reach a zeroing move in time.

So in a cursed win position, SF restricts to DTZ-optimal moves.

Suppose you have three moves leading to positions with DTZ=110 ply, DTZ=120 ply and DTZ=140 ply. Why should the engine consider playing anything else than the move leading to DTZ=110? The other two moves will not look "more natural": the problem of unnatural play only occurs for low values of DTZ. And the DTZ=110 clearly gives better chances at converting the draw into a win.

If SF would search all three moves, then which move will end up scoring best is essentially random. Without probing, SF is not able to see that a DTZ=110 move is better than a DTZ=120 move.

You can say what you want, but I can tell you that someone has spent a bit of time thinking about this. There are good reasons for all the choices.

I don't see any reason yet why you should pay any attention to the score at all if it is not a mate score. It seems always better to just play by DTZ if no mate could be detected.

The DTZ-optimal move might be Qb2 in order to force Kxb2. Of course if you can afford losing a queen, mate should not be too far away, but if thinking time is limited it might still not be found. And there are obviously more subtle situations where the DTZ-optimal move is too stupid for a human even to consider.

Btw, SF does not display DTZ scores. Too confusing for users. So to fix multipv, just randomly generate the missing moves and add them as 1-line PVs.

[hgm]

Suppose you have three moves leading to positions with DTZ=110 ply, DTZ=120 ply and DTZ=140 ply. Why should the engine consider playing anything else than the move leading to DTZ=110?

Why should an engine consider a move that gets it mated in 3 while it has one that keepsit a Pawn up? Because the user asks it to, obviously...

If SF would search all three moves, then which move will end up scoring best is essentially random. Without probing, SF is not able to see that a DTZ=110 move is better than a DTZ=120 move.

So the scoring is broken. If you think DTZ=110 is always better, the search should always return a score that is higher for the DTZ=110 move then for the DTZ=120 move. The only scores that should be equal to top it should come from moves that force better conversion, (i.e. to a sucessor EGT with a much smaller DTZ than for which the DTZ=110 move is going) and you already know there will be no conversions to be seen.

You can say what you want, but I can tell you that someone has spent a bit of time thinking about this. There are good reasons for all the choices.

Well, when the result embarrasingly sucks, they obviously did not think enough. A trivial solution to all these problems would be to score EGT probes first by game result, then by defending material, then by attacking material, and then by DTZ. Only when the root is in an EGT position, of course; you don't want these scores to be compared to the heuristic evaluation. (Which could be a problem when they represent cursed wins, when you score the probes somewhere between mate scores and the highest plausible heuristic evaluation.)