Stockfish Natural TB loses heavily to Stockfish master

[jhellis3]

Code: Select all

&& alpha > -VALUE_KNOWN_WIN // Stop probing if we have 
&& beta < VALUE_KNOWN_WIN&#41; // reached a won position. joster 

Although it is unlikely to make much difference, there is some imprecision there.

Instead you could use:

Code: Select all

&& !&#40;alpha => VALUE_KNOWN_WIN || beta <= -VALUE_KNOWN_WIN&#41;

[BBauer]

Thank you, I will try it.

Btw, testing has mostly be done with 5 piece syzyzgys.
I would expect most benefit from 6 piece syzyzgys.

Kind regards
Bernhard

[Laskos]

Hi Bernhard, you are using SF for analysis, right?

Did you try the early_mate modification?
https://github.com/official-stockfish/S ... /pull/1241
https://github.com/syzygy1/Stockfish/tree/early_mate

Do you use SF+TBs also for analysing positions with 6 pieces or less, so positions that are in the TBs?

Ronald, does early_mate2 achieve its goal?

Easy 5-men White Wins at the root


Stockfish Master: 500/500 Wins
Mean Win: 22.66 moves
Median Win: 19 moves

Stockfish Early_Mate2: 500/500 Wins
Mean Win: 22.93 moves
Median Win: 19 moves

Both close to optimal play.
Compare that to:

Texel (optimal DTM + DTZ50 play): 500/500 Wins
Mean Win: 18.55 moves
Median Win: 17 moves

Stockfish NTB: 496/500 Wins
Mean Win: 32.7 moves
Median Win: 26 moves

And now whether Early_Mate2 resolves faster Mates than Master:

1000 games, 500 as White and 500 as Black, in White easy 5-men Wins.

Total Mates resolved:
Stockfish Master: 12386
Stockfish Early_Mate2: 12324

Mates resolved on the first move of the game:
Stockfish Master: 322
Stockfish Early_Mate2: 323

Totally inconclusive whether Mates are resolved faster. I don't know about "naturalness" of play or the quantity of "weird" moves, I haven't looked carefully on the games, and I didn't studied the behavior in this sense with non-TB positions. ELO-wise the patch is completely neutral. But it seems neutral in resolving mates too (from TB root positions).

[BBauer]

Yes, I use SF for analysis.
I use my own version of SF with some mods.

Code: Select all

MaxDepth    = std::max(depth / ONE_PLY,MaxDepth);
    // Step 8. Null move search with verification search (is omitted in PV nodes)
    if (   !PvNode
	&&  depth       >= MaxDepth / NullMove * ONE_PLY     
	&&  eval >= beta
        && (ss->staticEval >= beta - 35 * (depth / ONE_PLY - 6) || depth >= 13 * ONE_PLY)
        &&  pos.non_pawn_material(pos.side_to_move()))
    {

This way I may use less nullmove pruning if I want by setting NullMove to a smaller value. Default is a big number.
Here an example
[D]8/8/4R1p1/1p1n1k2/8/5K2/1P6/8 w - - bm Rc6; id "Spike-Nebula TCEC Season 6, Stage 1b";
SF finds Rc6 very fast with 6 piece syzygys but takes a long time with 5 piece syzygys. I can not comment on internal use for syzygys. This could be improved but natural TBs looks useless to me. In fact it looks more like a threat.
Kind regards
Bernhard

[syzygy]

Ronald, does early_mate2 achieve its goal?

Easy 5-men White Wins at the root

There is no change in play if the root is a TB position. Basically, it just lets the regular SF search do the mate finding (but removes the non-winning moves from consideration).

Texel (optimal DTM + DTZ50 play): 500/500 Wins
Mean Win: 18.55 moves
Median Win: 17 moves

With DTM, Texel should win this.

And now whether Early_Mate2 resolves faster Mates than Master:

So early_mate is supposed to do this as well as SF master when starting from a root position in the TBs, but not better.

The only thing that early_mate changes from master is that it tries to find a mate in lines that enter the TB. Master always stops searching a branch of the tree once it enters the TB, thereby hiding a sometimes trivial mate from the search. Early_mate continues to search (but only if it has already proven the position to be a TB win, at least that is the idea).

So if a mate is hiding closely behind a TB win, SF master will not find it until the game reaches the TB win position on the board. Early_mate may find such mate before the game reaches the TB win position on the board (so "early" compared to SF master).

An example from an old thread (a not so trivial mate in 16):
[D]5r2/8/3k4/8/7R/P3R3/1P4K1/8 w - - 0 62
Early_mate with 6-piece TBs:

Code: Select all

info depth 8 seldepth 17 multipv 1 score cp 744 nodes 35724 nps 2381600 tbhits 526 time 15 pv h4d4 d6c7 e3c3 c7b6 d4b4 b6a5 c3c5 a5a6 c5c6 a6a7 c6c7 a7a8 b4b6 f8g8 g2f2
info depth 9 seldepth 22 multipv 1 score cp 13273 nodes 37313 nps 2487533 tbhits 564 time 15 pv h4d4 d6c7 e3c3 c7b6 d4b4 b6a5 c3c5 a5a6 c5c6 a6a7 c6c7 a7a8 c7a7 a8a7
...
info depth 48 seldepth 76 multipv 1 score mate 22 nodes 273052883 nps 3969772 hashfull 997 tbhits 14449648 time 68783 pv h4d4 d6c5 d4d2 f8g8 e3g3 g8e8 g3c3 c5b5 d2d5 b5b6 d5d4 e8g8 g2f3 g8f8 f3e4 f8e8 e4f5 e8h8 d4b4 b6a5 c3b3 h8f8 f5e5 f8e8 e5d6 e8h8 d6c7 h8h7 c7b8 h7h8 b8a7 h8h7 b4b7 a5a4 b3b6 h7b7 a7b7 a4a5 b7c6 a5a4 b6b7 a4a5 b7a7

Finding the TB win is instantaneous. Finding the mate takes a lot longer.

SF-master with TBs will not find a mate here, by design.

SF-master without TBs:

Code: Select all

info depth 36 seldepth 66 multipv 1 score mate 36 nodes 533322439 nps 4217988 hashfull 999 tbhits 0 time 126440 pv h4g4 d6c7 e3c3 c7d6 b2b4 f8f6 c3e3 d6c6 a3a4 f6d6 e3c3 c6b6 b4b5 b6a5 c3c6 d6d2 g2g1 d2d1 g1h2 d1h1 h2g3 h1h3 g3f2 h3f3 f2e1 f3e3 e1d2 e3d3 d2c1 d3d1 c1c2 d1d6 c6a6 d6a6 b5a6 a5a6 c2d2 a6a5 d2d3 a5b6 d3e2 b6c6 e2f3 c6c5 g4g5 c5d6 f3g2 d6e6 a4a5 e6f6 g5b5 f6e6 a5a6
info depth 37 seldepth 62 multipv 1 score mate 30 nodes 551799946 nps 4235069 hashfull 999 tbhits 0 time 130293 pv h4g4 d6d5 b2b4 f8c8 a3a4 c8c2 g2f3 c2c1 b4b5 c1f1 f3e2 f1b1 a4a5 b1c1 e2f3 c1f1 f3g2 f1b1 g4g3 b1b2 g2f3 b2b1 b5b6 d5c5 g3g6 b1f1 f3e2 f1h1 b6b7 h1h2 e2f3 h2h3 f3g2 h3h8 g6g4 h8d8 e3c3 c5b5 b7b8r d8b8 c3b3 b5c5 b3b8 c5d5 a5a6 d5e6 a6a7 e6f5 a7a8q f5g4 b8b4 g4f5

This was done on a machine under full load, so total node count is most relevant here.

Early_mate needs more depth than TB-less master, but at least in this position gets to that depth more quickly. In general, early_mate is probably slower than TB-less master in finding the mate. Probably because of how reductions work if the (TB win) scores found are really high.

SF-natural seems to get completely stuck searching at depth 23.

ELO-wise the patch is completely neutral. But it seems neutral in resolving mates too (from TB root positions).

This is good news

Admittedly, the term "early mate" suggests more than the patch actually does.

I may have a look at producing a patch that actively uses TBs in mate finding starting from a TB win, just to see what can be done.

[hgm]

I still don't understand why you would do this only for finding mates. Why not also keep searching for a more favorable conversion (gaining material, rather than sacrificing it)?

[Laskos]

Ronald, does early_mate2 achieve its goal?

Easy 5-men White Wins at the root

There is no change in play if the root is a TB position. Basically, it just lets the regular SF search do the mate finding (but removes the non-winning moves from consideration).

Texel (optimal DTM + DTZ50 play): 500/500 Wins
Mean Win: 18.55 moves
Median Win: 17 moves

With DTM, Texel should win this.

And now whether Early_Mate2 resolves faster Mates than Master:

So early_mate is supposed to do this as well as SF master when starting from a root position in the TBs, but not better.

The only thing that early_mate changes from master is that it tries to find a mate in lines that enter the TB. Master always stops searching a branch of the tree once it enters the TB, thereby hiding a sometimes trivial mate from the search. Early_mate continues to search (but only if it has already proven the position to be a TB win, at least that is the idea).

So if a mate is hiding closely behind a TB win, SF master will not find it until the game reaches the TB win position on the board. Early_mate may find such mate before the game reaches the TB win position on the board (so "early" compared to SF master).

An example from an old thread (a not so trivial mate in 16):
[D]5r2/8/3k4/8/7R/P3R3/1P4K1/8 w - - 0 62
Early_mate with 6-piece TBs:

Code: Select all

info depth 8 seldepth 17 multipv 1 score cp 744 nodes 35724 nps 2381600 tbhits 526 time 15 pv h4d4 d6c7 e3c3 c7b6 d4b4 b6a5 c3c5 a5a6 c5c6 a6a7 c6c7 a7a8 b4b6 f8g8 g2f2
info depth 9 seldepth 22 multipv 1 score cp 13273 nodes 37313 nps 2487533 tbhits 564 time 15 pv h4d4 d6c7 e3c3 c7b6 d4b4 b6a5 c3c5 a5a6 c5c6 a6a7 c6c7 a7a8 c7a7 a8a7
...
info depth 48 seldepth 76 multipv 1 score mate 22 nodes 273052883 nps 3969772 hashfull 997 tbhits 14449648 time 68783 pv h4d4 d6c5 d4d2 f8g8 e3g3 g8e8 g3c3 c5b5 d2d5 b5b6 d5d4 e8g8 g2f3 g8f8 f3e4 f8e8 e4f5 e8h8 d4b4 b6a5 c3b3 h8f8 f5e5 f8e8 e5d6 e8h8 d6c7 h8h7 c7b8 h7h8 b8a7 h8h7 b4b7 a5a4 b3b6 h7b7 a7b7 a4a5 b7c6 a5a4 b6b7 a4a5 b7a7

Finding the TB win is instantaneous. Finding the mate takes a lot longer.

SF-master with TBs will not find a mate here, by design.

SF-master without TBs:

Code: Select all

info depth 36 seldepth 66 multipv 1 score mate 36 nodes 533322439 nps 4217988 hashfull 999 tbhits 0 time 126440 pv h4g4 d6c7 e3c3 c7d6 b2b4 f8f6 c3e3 d6c6 a3a4 f6d6 e3c3 c6b6 b4b5 b6a5 c3c6 d6d2 g2g1 d2d1 g1h2 d1h1 h2g3 h1h3 g3f2 h3f3 f2e1 f3e3 e1d2 e3d3 d2c1 d3d1 c1c2 d1d6 c6a6 d6a6 b5a6 a5a6 c2d2 a6a5 d2d3 a5b6 d3e2 b6c6 e2f3 c6c5 g4g5 c5d6 f3g2 d6e6 a4a5 e6f6 g5b5 f6e6 a5a6
info depth 37 seldepth 62 multipv 1 score mate 30 nodes 551799946 nps 4235069 hashfull 999 tbhits 0 time 130293 pv h4g4 d6d5 b2b4 f8c8 a3a4 c8c2 g2f3 c2c1 b4b5 c1f1 f3e2 f1b1 a4a5 b1c1 e2f3 c1f1 f3g2 f1b1 g4g3 b1b2 g2f3 b2b1 b5b6 d5c5 g3g6 b1f1 f3e2 f1h1 b6b7 h1h2 e2f3 h2h3 f3g2 h3h8 g6g4 h8d8 e3c3 c5b5 b7b8r d8b8 c3b3 b5c5 b3b8 c5d5 a5a6 d5e6 a6a7 e6f5 a7a8q f5g4 b8b4 g4f5

This was done on a machine under full load, so total node count is most relevant here.

Early_mate needs more depth than TB-less master, but at least in this position gets to that depth more quickly. In general, early_mate is probably slower than TB-less master in finding the mate. Probably because of how reductions work if the (TB win) scores found are really high.

SF-natural seems to get completely stuck searching at depth 23.

ELO-wise the patch is completely neutral. But it seems neutral in resolving mates too (from TB root positions).

This is good news

Admittedly, the term "early mate" suggests more than the patch actually does.

I may have a look at producing a patch that actively uses TBs in mate finding starting from a TB win, just to see what can be done.

Ok, understood, might test it soon on one of my suites, I have to see which one is more sensitive. ELO-wise it was tested on non-TB as neutral, but was not tested on non-TB as a mate resolver compared to Master (both using 6-men).

[syzygy]

I still don't understand why you would do this only for finding mates. Why not also keep searching for a more favorable conversion (gaining material, rather than sacrificing it)?

Finding a more favorable conversion would require scoring TB win positions depending on, in particular, their material balance. For that you don't even need to search "behind" the conversion into the TBs.

It seems the two approaches are essentially independent of each other.

[Laskos]

Ok, understood, might test it soon on one of my suites, I have to see which one is more sensitive. ELO-wise it was tested on non-TB as neutral, but was not tested on non-TB as a mate resolver compared to Master (both using 6-men).

Tested, seems an excellent patch. Going to bed now. I took easy 6-men Wins, but only 5-men Syzygy, I guessed correctly that this combination will give sensible results. And it gave.

1000 games from this 6-men Wins with 5-men TBs at 0.25s per move (no time losses):

Score of SF_Master vs SF_Early_Mate2: 477 - 483 - 40 [0.497] 1000
ELO difference: -2.08 +/- 21.09
Finished match

ELO difference: -2.08 +/- 3.31 (pentanomial error margins)

Won games length:

Stockfish Master Wins:
Mean game length: 33.80 moves
Median game length: 30 moves

Stockfish Early_Mate2 Wins:
Mean game length: 25.28 moves
Median game length: 22 moves

Much shorter paths to Win for Early_Mate2 compared to Master.

Total Mates resolved from the start to the end of the games:
Stockfish Master: 4062
Stockfish Early_Mate2: 5090

Mates resolved in the first 5 moves of the game:
Stockfish Master: 10
Stockfish Early_Mate2: 48

Conclusive and very good results. Taking also into account that the patch is ELO-neutral. Must be committed.

[hgm]

Finding a more favorable conversion would require scoring TB win positions depending on, in particular, their material balance.

Well, that is quite trivial to do, and I don't see any possible downside.

For that you don't even need to search "behind" the conversion into the TBs.

I am not sure of that, as sometimes you might need a simplification or even a gain by forcing the opponent to trade. So that initially the material combination gets worse, but in the first move of that EGT you immediately convert back to a more favorable one.

It seems the two approaches are essentially independent of each other.

They seem the same to me. TB wins are mate scores with a long, undetermined DTM. When you find a very distant mate by finding a way to force a winning conversion, you just search on to see if you can find a closer mate. That can be an actual mate within the search horizon, or a closer undetermined mate by forcing conversion to a better (i.e. closer) TB win.