Oddity around depths 7-8 with Stockfish 6 & 7

[KWRegan]

This is one of what my large-scale data scripts tell me are thousands of examples, from the first tiebreak game between Magnus Carlsen and Yu Yangyi at last month's Qatar Masters:

[d]r1bq1rk1/1p3ppp/p1nbpn2/2pp4/3P4/2PBPNB1/PP1N1PPP/R2QK2R w KQ - 5 9

Stockfish 7, default settings (1 thread, 1pv, 16MB hash etc.; no real change if hash upped to 512MB), Win64 in Arena GUI:

Code: Select all

  1/1	00:00	      84	 42,000	+0.11	9.0-0
  2/2	00:00	     147	 73,500	+0.28	9.0-0 Bxg3
  3/3	00:00	     268	134,000	+0.31	9.dxc5 Bxc5 10.0-0
  4/4	00:00	     362	120,666	+0.81	9.dxc5 Bxg3 10.hxg3 a5
  5/5	00:00	     797	265,666	+0.43	9.Bxd6 Qxd6 10.dxc5 Qxc5 11.0-0
  6/6	00:00	   2,079	519,750	+0.22	9.0-0 Bxg3 10.hxg3 b6 11.Bc2 c4
  7/8	00:00	   3,664	732,800   +4.12	9.Bxd6 cxd4 10.Bxf8 dxe3 11.fxe3 Qxf8 12.0-0
  8/9	00:00	   5,863	837,571	+0.59	9.Bxd6 Qxd6 10.dxc5 Qd7 11.Nb3 Qe7 12.0-0 e5
 9/13	00:00	  17,411	916,368	+0.27	9.dxc5 Bxc5 10.0-0 Qe7 11.Ne5 Bb6 12.Nxc6 bxc6 13.e4
...

Anomaly: +4.12 9.Bxd6 cxd4 10.Bxf8 dxe3 11.fxe3 Qxf8 12.0-0

Stockfish 6, same settings

Code: Select all

  1/1	00:00	      86	   86,000	+0.27	9.0-0
  2/2	00:00	     150	  150,000	+0.42	9.0-0 Bxg3 10.hxg3 cxd4 11.exd4
  3/3	00:00	     272	  272,000	+0.40	9.Bxd6 Qxd6 10.0-0
  4/4	00:00	     473	  473,000	+0.44	9.dxc5 Bxc5 10.0-0 b6
  5/5	00:00	     762	  762,000	+0.55	9.dxc5 Bxg3 10.hxg3 b6 11.cxb6
  6/6	00:00	   1,053	1,053,000	+0.55	9.dxc5 Bxg3 10.hxg3 b6 11.cxb6 Qxb6
  7/8	00:00	   3,670	3,670,000	+4.22	9.Bxd6 cxd4 10.Bxf8 dxc3 11.bxc3 Qxf8 12.0-0
  8/9	00:00	   6,273   6,273,000	+4.29	9.Bxd6 Nxd4 10.Bxc5 Nxf3+ 11.Nxf3 Re8 12.0-0 b6 13.Ba3
 9/11	00:00	  12,332  12,332,000	+0.50	9.Bxd6 Qxd6 10.dxc5 Qe7 11.0-0 e5 12.e4 Bg4 13.exd5 Nxd5
...

Anomaly:
+4.22 9.Bxd6 cxd4 10.Bxf8 dxc3 11.bxc3 Qxf8 12.0-0
+4.29 9.Bxd6 Nxd4 10.Bxc5 Nxf3+ 11.Nxf3 Re8 12.0-0 b6 13.Ba3

Depth 8 strikes me as rather late to be blipping on an obvious recapture, one that was evaluated correctly at depth 5 (SF7) or 3 (SF6). The issue is absent in Stockfish 5 and all earlier versions I've tried, and absent in Komodo 9.3.

I will soon make a post on the computer science blog Gödel's Lost Letter (https://rjlipton.wordpress.com/) about a matter of larger interest. The relevance of low-depth values for predicting human actions is shown in the post https://rjlipton.wordpress.com/2015/10/ ... tisficing/ last October. I'm also simply curious---I don't think it's a hash-collision issue and (as the post will show) it shows up frequently at depths up through 8 but is better at 9 and evidently mostly gone by depth 10.

(I've posted this here rather than in the technical forum because it's not referring to code or algorithm strategy directly, though moving it there would be OK with me.)

[gladius]

Stockfish does some unsafe pruning in the PV which can lead to strange things like this at low depths. I don't like it, but I tried pretty hard to remove it in the past without success.

[BBauer]

I don't like it either.
It comes from search.cpp in Step 13.

Code: Select all

// Move count based pruning
          if (   depth < 16 * ONE_PLY
              && moveCount >= FutilityMoveCounts&#91;improving&#93;&#91;depth&#93;)
              continue;

should be:

Code: Select all

// Move count based pruning
	  if (   depth     >= 2* ONE_PLY       
     	      && depth     <  16* ONE_PLY
              && moveCount >= FutilityMoveCounts&#91;improving&#93;&#91;depth&#93;)
              continue;code&#93;

Now the oddity has vanished.
Kind regards
Bernhard

[hgm]

So what exactly causes the problem here? The patch you show would not affect pruning of the QxB recapture, which takes place at a remaining depth of 6 ply. So why does it refrain from that recapture at d=7, while it still recommended it at d=5? Does it imagine something terrible will happen near the leaves, when it plays Qxd6? What it imagines must be truly terrible, that it is willing to suffer a -4.12 score to avoid it. Only the loss of a Queen seems worse. Is the problem that two ply before the horizon the Queen can get attacked on d6, and that it prunes all evasions in the d=1 daughter, so it will be taken in QS?

This cannot be optimal. You might only see it in the root at low depth, but it must occur zillions of times near the leaves at any depth. It cannot be good that a substantial fraction of the sub-trees near the leaves are absolute garbage.

[BBauer]

I cannot follow your reasoning.

What I get after the modification is

Code: Select all

<--1&#58;readyok
<--1&#58;info depth 1 seldepth 1 multipv 1 score cp 10 nodes 84 nps 42000 tbhits 0 time 2 pv e1g1
<--1&#58;info depth 2 seldepth 2 multipv 1 score cp -16 nodes 328 nps 109333 tbhits 0 time 3 pv e1g1 c5c4 g3d6 d8d6
<--1&#58;info depth 3 seldepth 3 multipv 1 score cp 29 nodes 521 nps 173666 tbhits 0 time 3 pv d4c5 d6c5 e1g1
<--1&#58;info depth 4 seldepth 4 multipv 1 score cp 7 nodes 682 nps 170500 tbhits 0 time 4 pv d4c5 d6g3 h2g3 e6e5
<--1&#58;info depth 5 seldepth 5 multipv 1 score cp 41 nodes 994 nps 198800 tbhits 0 time 5 pv g3d6 d8d6 d4c5 d6c5 e1g1
<--1&#58;info depth 6 seldepth 6 multipv 1 score cp 13 nodes 1694 nps 242000 tbhits 0 time 7 pv g3d6 d8d6 e1g1 c5c4 d3c2 c8d7
<--1&#58;info depth 7 seldepth 7 multipv 1 score cp 18 nodes 2164 nps 270500 tbhits 0 time 8 pv g3d6 d8d6 e1g1 c5c4 d3c2 c8d7 e3e4
<--1&#58;info depth 8 seldepth 8 multipv 1 score cp 1 nodes 5037 nps 359785 tbhits 0 time 14 pv g3d6 d8d6 e1g1 c8d7 d3c2 e6e5 d4e5 c6e5
<--1&#58;info depth 9 seldepth 12 multipv 1 score cp 8 nodes 14881 nps 413361 tbhits 0 time 36 pv e1g1 d6g3 h2g3 c5c4 d3c2 c8d7 f3e5 h7h6 d2f3
<--1&#58;info depth 10 seldepth 13 multipv 1 score cp 20 nodes 36428 nps 467025 tbhits 0 time 78 pv d4c5 d6g3 h2g3 e6e5 e3e4 h7h6 d3c2 c8g4 e1g1 d5d4 d2c4
<--1&#58;info depth 11 seldepth 15 multipv 1 score cp 14 nodes 77171 nps 521425 tbhits 0 time 148 pv d4c5 d6c5 e1g1 c8d7 f1e1 c5e7 e3e4 d5e4 d2e4 f6e4 d3e4 f7f5
<--1&#58;info depth 12 seldepth 16 multipv 1 score cp 17 nodes 194181 nps 556392 tbhits 0 time 349 pv d4c5 d6c5 e1g1 c5e7 d1c2 h7h6 h2h3 e7d6 g3d6 d8d6 e3e4 c8d7 e4d5 e6d5 a1d1

And this looks more normal to me.
Kind regards
Bernhard

[hgm]

What line refutes 9... Qxd6 then, compared to losing Rook vs Pawn, without your patch at d=7?

[BBauer]

Perhaps Gary can answer your question.

Here we have a simple bug.
It makes no sense to do move count pruning at depth 1. Compare code.

Code: Select all

  // Futility and reductions lookup tables, initialized at startup
  int FutilityMoveCounts[2][16];  // [improving][depth]
  Depth Reductions[2][2][64][64]; // [pv][improving][depth][moveNumber]

Kind regards
Bernhard

[Evert]

Perhaps Gary can answer your question.

Here we have a simple bug.
It makes no sense to do move count pruning at depth 1.

Why not?
What makes the root node different from any other PV node in this respect?

Compare code.

Code: Select all

  // Futility and reductions lookup tables, initialized at startup
  int FutilityMoveCounts[2][16];  // [improving][depth]
  Depth Reductions[2][2][64][64]; // [pv][improving][depth][moveNumber]

I don't understand what point you're trying to make...?

[BBauer]

My point is: mcp should start at 2, not at 1.
If it starts at 1 we have odd behavior.
If it starts at 2 we have normal behavior.
This has been proven for the position in question.
Kind regrds
Bernhard

[Evert]

My point is: mcp should start at 2, not at 1.

So you said. But why?

If it starts at 1 we have odd behavior.
If it starts at 2 we have normal behavior.
This has been proven for the position in question.

Which says nothing about other positions. Fixing problems (or perceived problems) based on single positions is generally misleading and therefor dangerous. Especially if it isn't entirely clear why a certain change appears to correct the problem.