tt related search explosion?

[Sazgr]

Hi,
Recently I saw that on some of the moves in games that my engine plays, the node count from one depth to the next jumped by a lot. In the example search, the pv move did not change from depth 7 to depth 8, so I do not know why the node count jumped by 25x. My only thought is that it is related to the transposition table, as it does not happen when I clear the hash and search again, but I cannot pinpoint if further. Does anyone know what specifically in the TT might be causing this, and how I might fix it. Thanks in advance!

Code: Select all

info score cp 30 depth 1 nodes 51 nps 3591549 time 0 pv c1e3
info score cp 23 depth 2 nodes 175 nps 1697381 time 0 pv c1e3 d7c8
info score cp 22 depth 3 nodes 2262 nps 3548569 time 0 pv c1e3 h8d8 a1d1
info score cp 15 depth 4 nodes 4689 nps 3491775 time 1 pv c1e3 h8d8 a1d1 d7c8
info score cp -1 depth 5 nodes 26105 nps 3678190 time 7 pv c1e3 h8e8 c3b5 d7c8 b5d6
info score cp -4 depth 6 nodes 49389 nps 4177949 time 11 pv c1e3 h8e8 a1d1 a8d8 a2a4 d7c8
info score cp -1 depth 7 nodes 94778 nps 4951174 time 19 pv c1e3 h8e8 a1d1 a8d8 c3b5 d7c8 b5d6
info score cp -6 depth 8 nodes 2399117 nps 1930783 time 1242 pv c1e3 h8e8 h2h4 c6a5 h4h5 a5b3 a2b3 g6d3
info score cp -7 depth 9 nodes 25796608 nps 2578006 time 10006 pv c1e3 h8e8 a1d1 a8d8 h2h4 c6b4 h4h5 g6c2 b3c2
bestmove c1e3

[hgm]

One explanation could be that the TT is very helpful as long as the tree stays mainly the same, and is just expanded at the leaves. You then only need very few nodes compared to what you would need w ithout TT. This works up to depth 7, but at 8 ply the third move of the PV is superceded by h2h4. This move used to be refuted, but apparently the refutation held in the TT is no longer good enough, and no refutation can be found at all. So a new subtree will have to be constructed, without any guidance from the TT. That can take very long.

Does your engine use Internal Iterative Deepening? If not, then you know now why that is useful to have.

[Chessnut1071]

One explanation could be that the TT is very helpful as long as the tree stays mainly the same, and is just expanded at the leaves. You then only need very few nodes compared to what you would need w ithout TT. This works up to depth 7, but at 8 ply the third move of the PV is superceded by h2h4. This move used to be refuted, but apparently the refutation held in the TT is no longer good enough, and no refutation can be found at all. So a new subtree will have to be constructed, without any guidance from the TT. That can take very long.

Does your engine use Internal Iterative Deepening? If not, then you know now why that is useful to have.

Hi Muller. The issue may be due to the same problem you found in my tt last month-overlapping keys. I noticed a magnitude reduction in searches around that same ply level, the ply level which generates a large increase in hits. May want to check the code.

[Sazgr]

One explanation could be that the TT is very helpful as long as the tree stays mainly the same, and is just expanded at the leaves. You then only need very few nodes compared to what you would need w ithout TT. This works up to depth 7, but at 8 ply the third move of the PV is superceded by h2h4. This move used to be refuted, but apparently the refutation held in the TT is no longer good enough, and no refutation can be found at all. So a new subtree will have to be constructed, without any guidance from the TT. That can take very long.

Does your engine use Internal Iterative Deepening? If not, then you know now why that is useful to have.

Your explanation sounds very reasonable, and I think you have got the problem! My engine does not have iterative deepening, so I will look into it when I have some time.