Strange drop in evaluation

[Carbec]

Hello,

While testing the WinAtChess position #270, I noticed that the evaluation changed abruptly between depths. Not only the evaluation, but also the best move. In this position, the good one is Qg4.

Code: Select all

info score cp   57 depth  1 nodes           62 nps  62.000 time      1 pv e2e8
info score cp   57 depth  2 nodes          297 nps  99.000 time      3 pv e2e8 c8e8 e1e8
info score cp   57 depth  3 nodes        1.888 nps 377.600 time      5 pv e2e8 c8e8 e1e8 d7e8 d4a7
info score cp   43 depth  4 nodes        7.335 nps 611.250 time     12 pv e2e8 c8e8 e1e8 d7e8 d4a7 e8b5
info score cp   54 depth  5 nodes       29.670 nps 1.648.333 time     18 pv e2e8 c8e8 e1e8 d7e8 d4a7 e8e2 a7b8
info score cp   40 depth  6 nodes      109.942 nps 3.141.200 time     35 pv e2e8 c8e8 e1e8 d7e8 d4a7 e8c8 a7b6 c8c1
info score cp  149 depth  7 nodes      307.839 nps 4.216.972 time     73 pv d4g4 d7b5 a2a4 b5e2 e1e2 c8c1 f3e1 c1e1
info score cp  149 depth  8 nodes      612.989 nps 4.788.976 time    128 pv d4g4 d7b5 a2a4 b5e2 e1e2 c8c1 f3e1 c1e1 e2e1 e8e1
info score cp   51 depth  9 nodes    1.965.610 nps 5.552.570 time    354 pv e2e8 c8e8 d4a7 e8d8 a7b6 d8a8 a2a3 a8a4 h2h3 a4f4
info score cp   57 depth 10 nodes    5.404.244 nps 5.718.776 time    945 pv e2e8 c8e8 d4a7 e8c8 a7b6 h7h6 b6b3 b7b5 h2h4 c8c4 h4h5
info score cp  111 depth 11 nodes   12.567.084 nps 5.665.953 time  2.218 pv d4g4 d7b5 g4c4 b5d7 c4c7 d7a4 b2b3 a4b5 e2e8 c8e8 e1e8 b5e8 c7d6
info score cp  217 depth 12 nodes   20.873.073 nps 5.479.935 time  3.809 pv d4g4 d7b5 g4c4 b5d7 c4c7 d7b5 c7b7 b5e2 e1e2 c8c1 e2e1 c1e1 f3e1 e8e1
info score cp   83 depth 13 nodes   41.920.585 nps 5.486.269 time  7.641 pv d4g4 d7b5 g4c4 b5d7 c4c7 d7b5 e2e8 c8e8 e1e8 b5e8 c7b7 a7a5 b2b4 a5b4 b7b4
info score cp   89 depth 14 nodes   95.188.633 nps 5.536.476 time 17.193 pv d4g4 d7b5 g4c4 b5d7 c4c7 d7b5 e2e8 c8e8 e1e8 b5e8 c7b7 a7a5 b2b3 e8d8 b7b5 d8c7
info score cp  299 depth 15 nodes  174.172.717 nps 5.577.632 time 31.227 pv d4g4 d7b5 g4c4 b5d7 c4c7 d7b5 a2a4 b5a4 e2e4 e8f8 c7c8 a4e4 c8f8 g8f8 e1e4 f6b2
info score cp  310 depth 16 nodes  348.248.981 nps 5.590.050 time 62.298 pv d4g4 d7b5 g4c4 b5d7 c4c7 d7b5 a2a4 b5a4 e2e4 f6b2 c7c8 e8c8 e4a4 a7a6 f3d4 b2c3
info score cp  326 depth 17 nodes  814.273.718 nps 5.630.631 time 144.615 pv d4g4 d7b5 g4c4 b5d7 c4c7 d7b5 a2a4 b5a4 e2e4 f6b2 c7c8 e8c8 e4a4 a7a6 f3d4 b7b5 a4b4

Do you think it's normal ? or that I have a bug , perhaps the transposition table ? For information, I use the one from Sungorus, used also by Rodent.
Thanks for help)

Philippe

[JVMerlino]

[fen]2r1r1k1/pp1q1ppp/3p1b2/3P4/3Q4/5N2/PP2RPPP/4R1K1 w - -[/fen]

There's definitely something wrong, but it's hard to point out where.

Let's look at the PV displayed at depth 12 (note that this is BEFORE the score drop), which your engine scores at +2.17.
d4g4 d7b5 g4c4 b5d7 c4c7 d7b5 c7b7 b5e2 e1e2 c8c1 e2e1 c1e1 f3e1 e8e1
This sequence actually ends in checkmate...for Black! So the score should be -CHECKMATE.

The PV for depth 11, although containing inaccuracies, results in a reasonable score for the ending position. Same for depths 13 and 14. At depth 15, I would say that the engine has correctly solved the position. So something very strange is happening at depth 12, quite possibly due to a transposition table bug.

[Carbec]

Chapeau bas !!

There is indeed a bug, a very big one I didn't saw.
Thanks for this analyse, I have work to do.

Philippe

[flok]

Hmmm, my program Dog sees no checkmate at all:

Code: Select all

position fen 2r1r1k1/pp1q1ppp/3p1b2/3P4/3Q4/5N2/PP2RPPP/4R1K1 w - -
go depth 100
info depth 1 score cp 220 nodes 60 time 1 nps 60000 pv e2e8
info depth 2 score cp 220 nodes 130 time 1 nps 130000 pv e2e8 c8e8
# new ponder position (1/1)
# ponder search start
info depth 3 score cp 220 nodes 796 time 1 nps 796000 pv e2e8 c8e8 e1e8
info depth 4 score cp 220 nodes 4278 time 4 nps 1069500 pv e2e8 c8e8 e1e8 d7e8
info depth 5 score cp 220 nodes 5557 time 5 nps 1111400 pv e2e8 c8e8 e1e8 d7e8 d4a7
info depth 6 score cp 182 nodes 11507 time 7 nps 1643857 pv e2e8 c8e8 e1e8 d7e8 d4a7 e8b5
info depth 7 score cp 221 nodes 19557 time 10 nps 1955700 pv e2e8 c8e8 e1e8 d7e8 d4a7 f6b2 a7b7
info depth 8 score cp 171 nodes 30612 time 14 nps 2186571 pv e2e8 c8e8 e1e8 d7e8 d4a7 f6b2 a7b7 e8e2
info depth 9 score cp 179 nodes 84796 time 33 nps 2569575 pv e2e8 c8e8 d4a7 e8c8 f3d4 f6d4 a7d4 c8c2 e1e4
info depth 10 score cp 156 nodes 152810 time 58 nps 2634655 pv e2e8 c8e8 d4a7 e8c8 f3d4 f6d4 a7d4 c8c2 d4b6 h7h6
info depth 11 score cp 175 nodes 209542 time 78 nps 2686435 pv e2e8 c8e8 d4a7 e8c8 f3d4 f6d4 a7d4 c8c2 d4b6 h7h6 e1e4
info depth 12 score cp 159 nodes 537760 time 188 nps 2860425 pv e2e8 c8e8 d4a7 e8c8 f3d4 f6d4 a7d4 h7h6 d4e4 d7b5 b2b4 c8c3
info depth 13 score cp 165 nodes 1044220 time 359 nps 2908690 pv e2e8 c8e8 d4a7 e8c8 a7b6 h7h6 b2b4 c8c2 a2a3 f6c3 f3d4 c3d4 b6d4 d7b5
info depth 14 score cp 154 nodes 1387138 time 468 nps 2963970 pv e2e8 c8e8 d4a7 e8c8 a7b6 h7h6 b6b3 b7b5 f3d2 f6e5 h2h3 f7f5 d2f3 e5f4
info depth 15 score cp 148 nodes 2105905 time 690 nps 3052036 pv e2e8 c8e8 d4a7 e8c8 a7b6 h7h6 b6b3 b7b5 f3d2 f6e5 h2h4 e5f4 d2e4 c8c4 h4h5
info depth 16 score cp 149 nodes 2851715 time 911 nps 3130312 pv e2e8 c8e8 d4a7 e8c8 a7b6 h7h6 b6b3 b7b5 a2a3 c8c4 h2h3 g7g5 f3d2 c4a4 d2e4 f6d4
info depth 17 score cp 139 nodes 4144211 time 1285 nps 3225066 pv e2e8 c8e8 d4a7 e8e1 f3e1 h7h5 a7b6 d7e8 g1f1 e8e5 b6b7 e5h2 b7c8 g8h7 c8f5 h7h6 e1d3
info depth 18 score cp 128 nodes 5436999 time 1654 nps 690466 pv e2e8 c8e8 d4a7 e8e1 f3e1 h7h5 a7b6 d7e8 g1f1 e8e5 b6b7 e5h2 b7b8 g8h7 e1d3 h2h1 f1e2 h5h4
info depth 19 score cp 124 nodes 9308168 time 2756 nps 260607 pv e2e8 c8e8 d4a7 e8e1 f3e1 h7h6 a7a8 g8h7 a8a3 d7b5 a3d3 b5d3 e1d3 h7g6 g2g4 g6g5 h2h3 f6d4 g1f1 b7b5
info depth 20 score cp 123 nodes 11687747 time 3422 nps 905263 pv e2e8 c8e8 d4a7 e8e1 f3e1 h7h6 a7a8 g8h7 a8a3 d7b5 a3d3 b5d3 e1d3 h7g6 g2g4 g6g5 h2h3 f6d4 g1f1 b7b5
info depth 21 score cp 115 nodes 17791353 time 5163 nps 118435 pv e2e8 c8e8 d4a7 e8e1 f3e1 g7g5 b2b3 d7f5 a7b8 g8g7 b8d6 f6c3 d6e7 f5d5 e7e3 c3d2 e3d3 d5d3 e1d3 g7f6 d3c5
info depth 22 score cp 196 nodes 36354869 time 10320 nps 193326 pv d4g4 d7b5 g4c4 b5d7 c4c7 d7b5 e2e8 c8e8 e1e8 b5e8 b2b3 h7h6 c7d6 e8e2 h2h3 e2a2 d6b8 g8h7 b8b7 a2b1 g1h2 h7g6
info depth 23 score cp 212 nodes 47464094 time 13440 nps 16328 pv d4g4 d7b5 g4c4 b5d7 c4c7 d7b5 e2e8 c8e8 e1e8 b5e8 b2b3 h7h6 c7d6 e8e4 h2h4 e4b1 g1h2 b1a2 d6b8 g8h7 b8b7 h7g6 h2g1
info depth 24 score cp 550 nodes 181459281 time 51764 nps 20683 pv d4g4 d7b5 g4c4 b5d7 c4c7 d7b5 a2a4 b5a4 e2e4 f6d4 f3d4 h7h6 c7b7 c8b8 e4e8 b8e8 e1e8 a4e8 g1f1 e8a4 b7c8 g8h7 c8f5 h7g8
info depth 25 score cp 576 nodes 251123523 time 72262 nps 27890 pv d4g4 d7b5 g4c4 b5d7 c4c7 d7b5 a2a4 b5a4 e2e4 f6d4 f3d4 h7h6 c7b7 c8b8 e4e8 b8e8 e1e8 a4e8 g1f1 e8a4 b7c8 g8h7 c8f5 h7g8 d4c6
info depth 26 score cp 523 nodes 382006774 time 110470 nps 36658 pv d4g4 d7b5 g4c4 b5d7 c4c7 d7b5 a2a4 b5a4 e2e4 f6d4 f3d4 h7h6 c7b7 c8b8 e4e8 b8e8 e1e8 a4e8 d4f3 e8a4 b7b8 g8h7 h2h4 a4d1 g1h2 d1d5

[JVMerlino]

Hmmm, my program Dog sees no checkmate at all:

That's because there isn't one - at least, not one that can be easily found in a short search. Even the strongest engines will probably only give a score of about +5 after one minute.

The point was that this is a win for White, but the depth 12 PV ended in Black winning by checkmate, even though the score for that PV was in White's favor. Definitely pointing to something wrong....

[flok]

The point was that this is a win for White, but the depth 12 PV ended in Black winning by checkmate, even though the score for that PV was in White's favor. Definitely pointing to something wrong....

Aaah ok, I misread that. Thanks.

[Carbec]

Hello,

Thanks to JVMerlino, I found the problem. In the quiescence function, I didn't taken account if the side
is in check, or not. So now, if its in check, I generate all evasions moves. In the other case I generate
captures moves; and I initialyse stand_pat with evaluation.

The results are better. There is still one problem at depth 10, but its perhaps my evaluation.
In effect, I have only material + values from PieceSquareTables.

Code: Select all

info score cp   57 depth  1 nodes           66 nps  66.000 time      1 pv e2e8 c8e8 e1e8 d7e8 d4a7
info score cp   57 depth  2 nodes          531 nps 177.000 time      3 pv e2e8 c8e8 e1e8 d7e8 d4a7
info score cp   57 depth  3 nodes        2.344 nps 390.666 time      6 pv e2e8 c8e8 e1e8 d7e8 d4a7
info score cp   43 depth  4 nodes       10.077 nps 1.119.666 time      9 pv e2e8 c8e8 e1e8 d7e8 d4a7 e8b5
info score cp  432 depth  5 nodes       30.876 nps 1.816.235 time     17 pv d4g4 e8e2 g4d7 e2e1 f3e1 c8c1
info score cp   49 depth  6 nodes      126.819 nps 3.093.146 time     41 pv d4g4 d7b5 f3d4 f6d4 e2e8 c8e8 e1e8 b5e8 g4d4
info score cp   49 depth  7 nodes      270.764 nps 3.427.392 time     79 pv d4g4 d7b5 f3d4 f6d4 e2e8 c8e8 e1e8 b5e8 g4d4
info score cp   47 depth  8 nodes      760.195 nps 4.131.494 time    184 pv d4g4 d7b5 g4c4 b5d7 c4c7 e8d8 e2e8 d7e8 e1e8 d8e8 c7b7
info score cp  111 depth  9 nodes    1.578.510 nps 4.459.067 time    354 pv d4g4 d7b5 g4c4 b5d7 c4c7 d7a4 b2b3 a4b5 e2e8 c8e8 e1e8 b5e8 c7d6
info score cp   57 depth 10 nodes    5.284.347 nps 4.697.197 time  1.125 pv e2e8 c8e8 d4a7 e8c8 b2b3 h7h5 a7a4 b7b5 a4e4 d7g4 h2h3
info score cp  100 depth 11 nodes   11.478.990 nps 4.696.804 time  2.444 pv d4g4 d7b5 g4c4 b5d7 c4c7 d7b5 e2e8 c8e8 e1e8 b5e8 c7b7 f6e5 b7c7
info score cp  364 depth 12 nodes   19.725.033 nps 4.601.127 time  4.287 pv d4g4 d7b5 g4c4 b5d7 c4c7 d7b5 a2a4 b5a4 b2b3 e8f8 c7c8 a4b3
info score cp  382 depth 13 nodes   36.124.428 nps 4.632.524 time  7.798 pv d4g4 d7b5 g4c4 b5d7 c4c7 d7b5 a2a4 b5a4 b2b3 f6e5 c7c8 e8c8 b3a4
info score cp  299 depth 14 nodes   78.155.649 nps 4.687.557 time 16.673 pv d4g4 d7b5 g4c4 b5d7 c4c7 d7b5 a2a4 b5a4 e2e4 e8f8 c7c8 a4e4 c8f8 g8f8 e1e4 f6b2
info score cp  299 depth 15 nodes  151.901.595 nps 4.625.505 time 32.840 pv d4g4 d7b5 g4c4 b5d7 c4c7 d7b5 a2a4 b5a4 e2e4 e8f8 c7c8 a4e4 c8f8 g8f8 e1e4 f6b2
info score cp  310 depth 16 nodes  362.525.336 nps 4.661.865 time 77.764 pv d4g4 d7b5 g4c4 b5d7 c4c7 d7b5 a2a4 b5a4 e2e4 f6b2 c7c8 e8c8 e4a4 a7a6 f3d4 b2c3
info score cp  340 depth 17 nodes  817.078.646 nps 4.650.180 time 175.709 pv d4g4 d7b5 g4c4 b5d7 c4c7 d7b5 a2a4 b5a4 e2e4 f6b2 c7c8 e8c8 e4a4 a7a6 a4c4 c8d8 c4c7

I think that I have to work on the evaluation now. But I don't know how to advance.
That's work ahead !
Thanks

Philippe

[JVMerlino]

Good work! Glad my information was able to help you.