Stockfish Natural TB loses heavily to Stockfish master

[Dann Corbit]

{snip}
Basically the patch is neutral Elo-wise, but will show some more or shorter mates, is that correct?

For people who want to analyze games and positions (like me) this is what we want.

In fact, I would be more than willing to lose a few Elo for this capability.

When analyzing, you don't just want a good answer. You want the best possible answer, if possible.

[Laskos]

Another interesting thing about Final Natural TBs.

I have suites of theoretically won and drawn 5-men positions which occurred in real games. These are not specially selected hard positions, most of them are pretty trivial. I played at short TC (200ms per move) matches of 1000 games from them with SF Final NTB against SF master, to see at what rate Final NTB will fail to convert them. In fact I planned initially 4000 games matches, expecting an extremely low rate of failure.

Regular 5-men theoretical Wins:

Code: Select all

Games Completed = 1000 of 4000 (Avg game length = 8.897 sec)
Settings = RR/32MB/200ms per move/M 100000cp for 1000 moves, D 200000 moves/EPD:C:\LittleBlitzer\wins5_cleaned.epd(1560)
Time = 1313 sec elapsed, 3935 sec remaining
 1.  Stockfish 050917 64 BMI2 Master    	503.0/1000	500-494-6  	(L: m=494 t=0 i=0 a=0)	(D: r=2 i=1 f=3 s=0 a=0)	(tpm=191.8 d=42.49 nps=2563180)
 2.  Stockfish 050917 64 BMI2 Final NTB 	497.0/1000	494-500-6  	(L: m=500 t=0 i=0 a=0)	(D: r=2 i=1 f=3 s=0 a=0)	(tpm=192.2 d=39.30 nps=1387705)

At this time control, SF FNTB fails in 6 conversions out of 500 Wins against SF master.

Regular 5-men theoretical Draws:

Code: Select all

Games Completed = 1000 of 1000 (Avg game length = 7.358 sec)
Settings = RR/32MB/200ms per move/M 100000cp for 1000 moves, D 200000 moves/EPD:C:\LittleBlitzer\Draws5_cleaned.epd(9783)
Time = 1093 sec elapsed, 0 sec remaining
 1.  Stockfish 050917 64 BMI2 Master    	505.5/1000	11-0-989  	(L: m=0 t=0 i=0 a=0)	(D: r=140 i=589 f=256 s=4 a=0)	(tpm=206.5 d=22.25 nps=2282356)
 2.  Stockfish 050917 64 BMI2 Final NTB 	494.5/1000	0-11-989  	(L: m=11 t=0 i=0 a=0)	(D: r=140 i=589 f=256 s=4 a=0)	(tpm=44.8 d=124.33 nps=506748)

At this time control, SF FNTB fails in 11 conversions out of 1000 Draws against SF master.

I will re-test at more reasonable 400ms per move, expecting a lower rate of failure. Also, an estimation of 10% can be made about games decided by TBs at root, so the ELO regression of SF Final NTB can be estimated at below 0.5 ELO points from regular openings like 2moves_v1.epd. It will most probably pass the regression test, and will be accepted.

[syzygy]

I left overnight a 4000 games match with adjudication condition: Win if the score is above 13000cp for 1 move (almost all, if not all such scores, are TB wins with Stockfish Syzygy). It confirms your theory, the depth difference is insignificant now (0.03 plies, well within error margins):

Code: Select all

Games Completed = 4000 of 4000 (Avg game length = 31.061 sec)
Settings = RR/32MB/15000ms+150ms/M 13000cp for 1 moves, D 200000 moves/EPD:C:\LittleBlitzer\Disbalance2.epd(1079)
Time = 18029 sec elapsed, 0 sec remaining
 1.  Stockfish 030917 64 BMI2 Master     	2005.5/4000	903-892-2205  	(L: m=0 t=0 i=0 a=892)	(D: r=785 i=842 f=546 s=32 a=0)	(tpm=394.9 d=20.93 nps=1999492)
 2.  Stockfish 030917 64 BMI2 Easy_Mate2 	1994.5/4000	892-903-2205  	(L: m=0 t=0 i=0 a=903)	(D: r=785 i=842 f=546 s=32 a=0)	(tpm=397.9 d=20.90 nps=1996020)

Thanks! Good to see the theory confirmed.

As this adjudication rule does not change the outcome of the games, because both engines (Master and Easy_Mate2) play perfectly after TB win was found, I combined all my 4 runs in 8000 games match, and the total result for Easy_Mate2 is -1.8 +/- 2.9 ELO points (pentanomial error margins), completely inconclusive even from this very sensitive endgame suite. Basically the patch is neutral Elo-wise, but will show some more or shorter mates, is that correct?

That is correct.

Extreme example:
[D]kb6/b7/8/8/8/3K1p2/8/1R1B4 w - - 0 1

Stockfish-master:

Code: Select all

info depth 1 seldepth 2 multipv 1 score cp 13279 nodes 22 nps 11000 tbhits 2 time 2 pv d1f3
info depth 2 seldepth 2 multipv 1 score cp 13279 nodes 45 nps 15000 tbhits 4 time 3 pv d1f3
info depth 3 seldepth 2 multipv 1 score cp 13279 nodes 64 nps 21333 tbhits 5 time 3 pv d1f3
info depth 4 seldepth 2 multipv 1 score cp 13279 nodes 83 nps 27666 tbhits 6 time 3 pv d1f3
info depth 5 seldepth 2 multipv 1 score cp 13279 nodes 102 nps 34000 tbhits 7 time 3 pv d1f3
...
info depth 127 seldepth 2 multipv 1 score cp 13279 nodes 90711881 nps 2955073 hashfull 999 tbhits 3487429 time 30697 pv d1f3

Stockfish-early_mate:

Code: Select all

info depth 1 seldepth 2 multipv 1 score mate 1 nodes 22 nps 11000 tbhits 2 time 2 pv d1f3
info depth 2 seldepth 2 multipv 1 score mate 1 nodes 41 nps 20500 tbhits 3 time 2 pv d1f3
info depth 3 seldepth 2 multipv 1 score mate 1 nodes 60 nps 30000 tbhits 4 time 2 pv d1f3
info depth 4 seldepth 2 multipv 1 score mate 1 nodes 79 nps 39500 tbhits 5 time 2 pv d1f3
info depth 5 seldepth 2 multipv 1 score mate 1 nodes 98 nps 49000 tbhits 6 time 2 pv d1f3
...
info depth 127 seldepth 2 multipv 1 score mate 1 nodes 2416 nps 402666 tbhits 128 time 6 pv d1f3

[Laskos]

Another interesting thing about Final Natural TBs.

I have suites of theoretically won and drawn 5-men positions which occurred in real games. These are not specially selected hard positions, most of them are pretty trivial. I played at short TC (200ms per move) matches of 1000 games from them with SF Final NTB against SF master, to see at what rate Final NTB will fail to convert them. In fact I planned initially 4000 games matches, expecting an extremely low rate of failure.

Regular 5-men theoretical Wins:

Code: Select all

Regular 5-men Wins

Games Completed = 1000 of 4000 (Avg game length = 8.897 sec)
Settings = RR/32MB/200ms per move/M 100000cp for 1000 moves, D 200000 moves/EPD:C:\LittleBlitzer\wins5_cleaned.epd(1560)
Time = 1313 sec elapsed, 3935 sec remaining
 1.  Stockfish 050917 64 BMI2 Master    	503.0/1000	500-494-6  	(L: m=494 t=0 i=0 a=0)	(D: r=2 i=1 f=3 s=0 a=0)	(tpm=191.8 d=42.49 nps=2563180)
 2.  Stockfish 050917 64 BMI2 Final NTB 	497.0/1000	494-500-6  	(L: m=500 t=0 i=0 a=0)	(D: r=2 i=1 f=3 s=0 a=0)	(tpm=192.2 d=39.30 nps=1387705)

At this time control, SF FNTB fails in 6 conversions out of 500 Wins against SF master.

Regular 5-men theoretical Draws:

Code: Select all

Games Completed = 1000 of 1000 (Avg game length = 7.358 sec)
Settings = RR/32MB/200ms per move/M 100000cp for 1000 moves, D 200000 moves/EPD:C:\LittleBlitzer\Draws5_cleaned.epd(9783)
Time = 1093 sec elapsed, 0 sec remaining
 1.  Stockfish 050917 64 BMI2 Master    	505.5/1000	11-0-989  	(L: m=0 t=0 i=0 a=0)	(D: r=140 i=589 f=256 s=4 a=0)	(tpm=206.5 d=22.25 nps=2282356)
 2.  Stockfish 050917 64 BMI2 Final NTB 	494.5/1000	0-11-989  	(L: m=11 t=0 i=0 a=0)	(D: r=140 i=589 f=256 s=4 a=0)	(tpm=44.8 d=124.33 nps=506748)

At this time control, SF FNTB fails in 11 conversions out of 1000 Draws against SF master.

I will re-test at more reasonable 400ms per move, expecting a lower rate of failure. Also, an estimation of 10% can be made about games decided by TBs at root, so the ELO regression of SF Final NTB can be estimated at below 0.5 ELO points from regular openings like 2moves_v1.epd. It will most probably pass the regression test, and will be accepted.

At double, now reasonable time control 0.4s per move, the results are similar:

Code: Select all

Regular 5-men Wins:

Games Completed = 1000 of 1000 (Avg game length = 17.890 sec)
Settings = RR/32MB/400ms per move/M 100000cp for 1000 moves, D 200000 moves/EPD:C:\LittleBlitzer\Wins5_cleaned.epd(1560)
Time = 2624 sec elapsed, 0 sec remaining
 1.  Stockfish 050917 64 BMI2 Master       502.5/1000	500-495-5  	(L: m=495 t=0 i=0 a=0)	(D: r=0 i=1 f=4 s=0 a=0)	(tpm=370.1 d=45.89 nps=2620688)
 2.  Stockfish 050917 64 BMI2 Final NTB 	497.5/1000	495-500-5  	(L: m=500 t=0 i=0 a=0)	(D: r=0 i=1 f=4 s=0 a=0)	(tpm=368.0 d=43.20 nps=1413959)


Regular 5-men Draws:

Games Completed = 1000 of 1000 (Avg game length = 13.620 sec)
Settings = RR/32MB/400ms per move/M 100000cp for 1000 moves, D 200000 moves/EPD:C:\LittleBlitzer\Draws5_cleaned.epd(9783)
Time = 1996 sec elapsed, 0 sec remaining
 1.  Stockfish 050917 64 BMI2 Master    	509.0/1000	18-0-982  	(L: m=0 t=0 i=0 a=0)	(D: r=134 i=594 f=251 s=3 a=0)	(tpm=403.7 d=26.09 nps=2367293)
 2.  Stockfish 050917 64 BMI2 Final NTB 	491.0/1000	0-18-982  	(L: m=18 t=0 i=0 a=0)  (D: r=134 i=594 f=251 s=3 a=0)	(tpm=55.7 d=125.21 nps=516931)

5 misses out 500 regular 5-men Wins at the root, 18 misses out of 1000 regular 5-men Draws at the root for Stockfish Final Natural TB. So, expect some 1-2% rate of blunders in root TB positions with Natural TB enabled. Would be fun if it happens at TCEC.

[Dann Corbit]

{snip}
So, expect some 1-2% rate of blunders in root TB positions with Natural TB enabled. Would be fun if it happens at TCEC.

When given fairly long time control with a huge pile of cores, I think that blunders will be less probable.

I did not test it, but I suspect if you put probable blunder positions to the test with (perhaps) 8 cores at 10 minutes per move, that zero blunders will be made in a thousand trials.

The problem is that this test takes a lot of patience, and I guess nobody has the patience to do it.

[syzygy]

Would be fun if it happens at TCEC.

If I have understood correctly, TCEC will use 6-piece TB adjudication (respecting the 50-move rule this time). So SF may win games or hold draws it would not actually be able to win or hold.

[syzygy]

When given fairly long time control with a huge pile of cores, I think that blunders will be less probable.

Perhaps, but the important point is that these blunders are entirely unnecessary with a proper design.

[Dann Corbit]

When given fairly long time control with a huge pile of cores, I think that blunders will be less probable.

Perhaps, but the important point is that these blunders are entirely unnecessary with a proper design.

Then why not post a patch?

It seems to me, that this technique, properly implemented, is what every user would want.

Consider all the space wasted with a complete set of Nalimov files. If we can get a similar effect using Syzygy tables with a small tweak to the code, then think of all the space that can be saved by getting rid of the Nalimov tables.

Multiply it by some thousands for all of the different users that benefit.

[syzygy]

When given fairly long time control with a huge pile of cores, I think that blunders will be less probable.

Perhaps, but the important point is that these blunders are entirely unnecessary with a proper design.

Then why not post a patch?

This one?
https://github.com/official-stockfish/S ... /pull/1241

It seems to me, that this technique, properly implemented, is what every user would want.

See my "early mate" patch.

With this patch, SF with TBs may still take longer than SF without TBs to find a particular mate, but at least the most blatant cases of hidden mates are removed. (Of course SF with TBs, with or without the patch, will resolve the position as a TB win much earlier than SF without TBs.)

Importantly, my patch improves the cosmetics without diminishing SF's objective TB performance. And it is simple.

Consider all the space wasted with a complete set of Nalimov files. If we can get a similar effect using Syzygy tables with a small tweak to the code, then think of all the space that can be saved by getting rid of the Nalimov tables.

Multiply it by some thousands for all of the different users that benefit.

The patches we are discussing here are not going to approach what DTM tables can do in terms of mate finding.

[mcostalba]

At this time control, SF FNTB fails in 11 conversions out of 1000 Draws against SF master.

Thanks Kay for testing NTB!

Could you please post the pgn of some game where SF fails to keep the draw? This should not happen. Never.

Instead converting the win is another story. Let me clarify.

SF NTB is always able to convert a win, but because of the way it is designed, finding the winning move is not immediate. We are talking of few seconds, not hours. It is very difficult to give a general rule but you can assume that within 1 minute of search it is able to find anything that there is to find in a position (but in the most cases we are talking of just fractions of a second).

Nevertheless when I read "a reasonable 400ms per move" and then in the same line "expect blunder at TCEC" and "I use it for analysis" I understand to make people change their minds is mission impossible.

So I will add an UCI option "Natural TB" by which users can toggle between natural and traditional behavior. This of course is just a placebo knob, but it takes me much less to add it than to convince people otherwise. Of course during real analysis I expect Natural TB to be always enabled because people wants to see good analysis lines, in multi PV, no odd sacrifices and with proper scores to understand the difference between competing PV lines (and of course wants to see a mate if engine finds it): all things that Natural TB does....well...naturally

One last thing. I have further improved the way DTZ is able to steer the engine in finding the winning line. I have done it in a way to preserve all the good properties of Natural TB, but now winning line is found on average in much shorter time.

I have tested it on your 2 reported games with 5-men where SF failed to convert the wins. Now it does.

Code: Select all

[Event "?"]
[Site "?"]
[Date "2017.09.06"]
[Round "?"]
[White "stockfish NTB"]
[Black "stockfish_ms"]
[Result "1-0"]
[FEN "8/8/2K5/8/3k1B2/8/8/n2B4 w - - 0 1"]
[PlyCount "81"]
[SetUp "1"]
[TimeControl "60+0.1"]

1. Ba4 {+4.93/27 2.0s} Kc4 {-132.79/1 0.001s} 2. Be5 {+5.03/24 0.46s}
Nb3 {-132.79/1 0s} 3. Bb5+ {+5.09/25 0.54s} Kb4 {-132.79/1 0s}
4. Bd6+ {+5.04/24 0.18s} Kc3 {-132.79/1 0s} 5. Kd5 {+5.09/31 0.67s}
Nd2 {-132.79/31 1.6s} 6. Be5+ {+5.17/29 1.3s} Kc2 {-132.79/1 0s}
7. Bf6 {+5.17/33 0.84s} Kb3 {-132.79/1 0s} 8. Bd3 {+5.20/30 0.70s}
Kb4 {-132.79/1 0s} 9. Be5 {+5.20/32 0.87s} Nb3 {-132.79/1 0s}
10. Bg3 {+5.28/32 0.50s} Na1 {-132.79/1 0s} 11. Be1+ {+5.92/30 1.2s}
Kb3 {-132.79/1 0s} 12. Bd2 {+5.92/32 0.20s} Nc2 {-132.79/37 1.3s}
13. Be2 {+5.92/41 0.95s} Nb4+ {-132.79/38 0.26s} 14. Kc5 {+5.92/40 0.34s}
Na2 {-132.79/48 1.0s} 15. Bd1+ {+132.61/34 2.7s} Kb2 {-132.79/40 1.2s}
16. Kc4 {+132.60/33 0.97s} Nc1 {-132.79/46 1.8s} 17. Bc3+ {+132.62/29 2.7s}
Ka3 {-132.79/51 1.2s} 18. Bb4+ {+131.22/33 2.4s} Kb2 {-132.79/48 0.28s}
19. Bc3+ {+132.40/33 0.67s} Ka3 {-132.79/55 1.1s} 20. Be5 {+132.30/25 0.17s}
Na2 {-132.79/53 7.3s} 21. Bg4 {+132.45/29 0.64s} Nc1 {-M60/37 1.9s}
22. Bf3 {+132.37/38 1.1s} Nb3 {-M54/39 1.0s} 23. Kc3 {+M45/34 0.54s}
Nc1 {-M52/36 0.18s} 24. Kc2 {+M41/35 0.22s} Na2 {-M42/38 1.5s}
25. Kb1 {+M39/40 0.85s} Kb3 {-M40/42 0.92s} 26. Bd5+ {+M37/38 0.21s}
Ka3 {-M38/40 0.22s} 27. Bxa2 {+M33/40 0.89s} Kb4 {-M36/41 1.0s}
28. Bd4 {+M31/37 0.20s} Kb5 {-M26/42 1.1s} 29. Bd5 {+M29/39 0.85s}
Kb4 {-M24/44 1.0s} 30. Kb2 {+M23/40 1.2s} Kb5 {-M22/45 1.2s}
31. Kc3 {+M21/40 1.2s} Ka6 {-M20/41 0.20s} 32. Kc4 {+M19/37 0.19s}
Ka5 {-M18/1 0s} 33. Bb7 {+M17/38 1.1s} Ka4 {-M16/1 0s} 34. Be5 {+M15/35 0.23s}
Ka3 {-M14/41 0.97s} 35. Bc6 {+M13/38 0.83s} Ka2 {-M12/1 0s}
36. Kc3 {+M11/36 0.18s} Ka1 {-M10/52 0.98s} 37. Kb3+ {+M9/48 0.85s}
Kb1 {-132.79/1 0s} 38. Bf3 {+M7/61 0.17s} Kc1 {-M6/1 0s}
39. Bf4+ {+M5/127 0.12s} Kb1 {-M4/1 0s} 40. Be4+ {+M3/127 0.010s} Ka1 {-M2/1 0s}
41. Be5# {+M1/127 0.006s, White mates} 1-0

[Event "?"]
[Site "?"]
[Date "2017.09.06"]
[Round "?"]
[White "stockfish NTB"]
[Black "stockfish_ms"]
[Result "1-0"]
[FEN "5n2/8/8/8/5K2/1B6/2N5/6k1 w - - 0 1"]
[PlyCount "85"]
[SetUp "1"]
[TimeControl "60+0.1"]

1. Nb4 {+0.19/39 1.3s} Nd7 {-132.79/1 0.001s} 2. Nd3 {+0.19/24 0.14s}
Nb6 {-132.79/1 0.001s} 3. Be6 {+0.26/37 0.98s} Kg2 {-132.79/1 0s}
4. Kg4 {+0.30/34 0.16s} Na4 {-132.79/1 0s} 5. Bd5+ {+0.30/35 0.93s}
Kg1 {-132.79/1 0s} 6. Kf3 {+0.30/30 0.19s} Nc3 {-132.79/1 0s}
7. Bc6 {+0.31/38 0.76s} Kf1 {-132.79/1 0s} 8. Bd7 {+0.33/37 0.37s}
Kg1 {-132.79/1 0s} 9. Bc6 {+0.33/44 0.89s} Kf1 {-132.79/1 0s}
10. Bd7 {+0.33/37 0.19s} Kg1 {-132.79/1 0s} 11. Nb4 {+0.34/42 0.81s}
Nb1 {-132.79/23 0.82s} 12. Ke2 {+0.40/43 0.67s} Nc3+ {-132.79/1 0s}
13. Ke3 {+0.40/39 0.33s} Kf1 {-132.79/1 0s} 14. Nc2 {+42.25/35 1.7s}
Nd5+ {-132.79/1 0s} 15. Kf3 {+42.25/31 0.18s} Kg1 {-132.79/1 0s}
16. Bh3 {+41.71/40 2.0s} Kh2 {-132.79/1 0.001s} 17. Be6 {+0.41/42 1.8s}
Nc7 {-132.79/1 0s} 18. Bd7 {+0.38/31 0.38s} Nd5 {-132.79/1 0s}
19. Be6 {+0.37/39 1.3s} Nc7 {-132.79/1 0s} 20. Bd7 {+4.70/42 2.1s}
Nd5 {-132.79/1 0s} 21. Nd4 {+4.70/31 0.20s} Nf6 {-132.79/1 0s}
22. Bf5 {+4.70/37 1.3s} Ne8 {-132.79/30 2.5s} 23. Nb5 {+132.57/28 0.92s}
Ng7 {-132.79/1 0s} 24. Bd7 {+132.53/38 1.7s} Nh5 {-132.79/1 0.004s}
25. Nc3 {+132.61/34 2.7s} Nf6 {-132.79/1 0s} 26. Be6 {+132.50/37 2.1s}
Ne8 {-132.79/1 0s} 27. Ne4 {+132.55/31 0.19s} Nc7 {-132.79/1 0s}
28. Bf5 {+132.58/33 6.0s} Kg1 {-132.79/1 0s} 29. Nd2 {+132.53/35 1.2s}
Nd5 {-132.79/42 2.7s} 30. Kg3 {+132.59/36 3.9s} Nf6 {-132.79/1 0s}
31. Bg6 {+M35/34 0.46s} Nd5 {-132.79/1 0.004s} 32. Bd3 {+M27/43 0.62s}
Nf6 {-132.79/1 0s} 33. Nf3+ {+M25/44 0.16s} Kh1 {-132.79/1 0s}
34. Ng5 {+M23/48 0.72s} Nh5+ {-132.79/1 0.003s} 35. Kf2 {+M17/50 0.15s}
Kh2 {-132.79/1 0.003s} 36. Nf3+ {+M15/71 0.64s} Kh1 {-M14/77 2.8s}
37. Bf1 {+M13/72 0.13s} Nf4 {-M12/78 0.33s} 38. Ne5 {+M11/84 0.67s}
Kh2 {-M10/88 1.5s} 39. Ng4+ {+M9/84 0.18s} Kh1 {-M8/1 0s} 40. Ba6 {+M7/89 0.86s}
Ng2 {-M6/93 1.7s} 41. Bd3 {+M5/127 0.19s} Ne1 {-M4/95 1.7s}
42. Be4+ {+M3/127 0.036s} Nf3 {-M2/127 0.069s}
43. Bxf3# {+M1/127 0.004s, White mates} 1-0