Test positions that make you tear your hair out

[JVMerlino]

I'm well aware that trying to make your engine solve a single test position more quickly is a great way to lose elo. So no need to bring that up.

But this position really confounds Myrddin. I tried fifteen other engines in Myrddin's elo range (~2600 CCRL), and all but two found h5 in less than ten seconds, and most of those found it in less than two seconds. Myrddin, however, with eight cores, needs almost 15 MINUTES! And I cannot figure out why. I've tried the obvious things, like turning off null move, increasing/decreasing the aggressiveness of reductions, etc., etc. But Myrddin will only select either cxd5 or Qe3, with a score of about +0.7 for White. If I force h5, Myrddin sees the win in less than two seconds, with the score jumping to about +4.

[fen]r5n1/p2q3k/4b2p/3pB3/2PP1QpP/8/PP4P1/5RK1 w - - 1[/fen]

Not only am I asking for your thoughts on why this is happening, but also if you could share any test positions that are the bane of your engine, I would feel better about it.

jm

[JoAnnP38]

The dev version of Pedantic is also in the 2600s and has not found h5 after over an hour of search so it's not just your engine. What depth did you get to before discovering it? I could see that at depth 32, h5 was going to be selected, but so many nodes where having to be explored due to the change in PV my program never finished the search before I aborted the analysis after at least an hour had transpired. Switching to h5 blew out my aspiration window three consecutive times, so I had to re-search depth 32 multiple times and each time looking at over a billion nodes. It's a really tough position.

[Mike Sherwin]

SF takes to ply 6 to determine that h4h5 is best. However, SF takes to ply 13 to see that it is +7.57.
That tells me that it has just as much to do with the evaluation as it does with tactics. That means that h4h5 is not pruned as heavily in SF as it is in other engines.

[JVMerlino]

The dev version of Pedantic is also in the 2600s and has not found h5 after over an hour of search so it's not just your engine. What depth did you get to before discovering it? I could see that at depth 32, h5 was going to be selected, but so many nodes where having to be explored due to the change in PV my program never finished the search before I aborted the analysis after at least an hour had transpired. Switching to h5 blew out my aspiration window three consecutive times, so I had to re-search depth 32 multiple times and each time looking at over a billion nodes. It's a really tough position.

With eight cores it started to fail high at depth 25, but this is a bit misleading because I have some of the cores "look ahead" by searching deeper than the primary search core. It went through the two aspiration windows, and after that I switch to full-width window. I didn't bother to wait for it to return.

[JoAnnP38]

SF takes to ply 6 to determine that h4h5 is best. However, SF takes to ply 13 to see that it is +7.57.
That tells me that it has just as much to do with the evaluation as it does with tactics. That means that h4h5 is not pruned as heavily in SF as it is in other engines.

h4h5 is selected as the best move at depth 6 by Pedantic as being marginally better than whatever was selected at depth 5, but it was quickly discarded at depth 7.

[JVMerlino]

As always with chess engines, I discovered a way to make Myrddin find h5 in less than five seconds, but it makes it perform worse on almost every other test position I have. In the logic for whether or not a move is subject to LMR, I had this line:

Code: Select all

if (EvalPly > 1) // not at the root

So there would be no reductions at the root. Simple enough. I changed that number to 2, and bingo.

Code: Select all

16     17    351      8688177 f4e3 e6f5 c4d5 a8f8 e3a3 f8f7 d5d6 g8f6 a3b3 h7g6 f1c1 d7b7 b3b7 f7b7 b2b3 f6d5 (2471 KNPS)
16     38    471     11696223 h4h5! (2478 KNPS)
16     54    496     12390606 h4h5! (2493 KNPS)
16    362    711     17769545 h4h5 a8e8 f4f2 e6f5 f2f5 d7f5 f1f5 g8e7 f5f7 h7g8 f7g7 g8f8 c4d5 e7d5 g7a7 e8e6 g1f2 e6b6 b2b3 b6c6 f2e2 (2499 KNPS)

Sigh...

[Joerg Oster]

As always with chess engines, I discovered a way to make Myrddin find h5 in less than five seconds, but it makes it perform worse on almost every other test position I have. In the logic for whether or not a move is subject to LMR, I had this line:

Code: Select all

if (EvalPly > 1) // not at the root

So there would be no reductions at the root. Simple enough. I changed that number to 2, and bingo.

Code: Select all

16     17    351      8688177 f4e3 e6f5 c4d5 a8f8 e3a3 f8f7 d5d6 g8f6 a3b3 h7g6 f1c1 d7b7 b3b7 f7b7 b2b3 f6d5 (2471 KNPS)
16     38    471     11696223 h4h5! (2478 KNPS)
16     54    496     12390606 h4h5! (2493 KNPS)
16    362    711     17769545 h4h5 a8e8 f4f2 e6f5 f2f5 d7f5 f1f5 g8e7 f5f7 h7g8 f7g7 g8f8 c4d5 e7d5 g7a7 e8e6 g1f2 e6b6 b2b3 b6c6 f2e2 (2499 KNPS)

Sigh...

What about reducing the reduction for pawn pushes, or only for advanced pawn pushes?
Maybe this helps.

[Pio]

I'm well aware that trying to make your engine solve a single test position more quickly is a great way to lose elo. So no need to bring that up.

But this position really confounds Myrddin. I tried fifteen other engines in Myrddin's elo range (~2600 CCRL), and all but two found h5 in less than ten seconds, and most of those found it in less than two seconds. Myrddin, however, with eight cores, needs almost 15 MINUTES! And I cannot figure out why. I've tried the obvious things, like turning off null move, increasing/decreasing the aggressiveness of reductions, etc., etc. But Myrddin will only select either cxd5 or Qe3, with a score of about +0.7 for White. If I force h5, Myrddin sees the win in less than two seconds, with the score jumping to about +4.

[fen]r5n1/p2q3k/4b2p/3pB3/2PP1QpP/8/PP4P1/5RK1 w - - 1[/fen]

Not only am I asking for your thoughts on why this is happening, but also if you could share any test positions that are the bane of your engine, I would feel better about it.

jm

I would consider the pawn push because the position is almost zugzwang and if you don’t push the pawn the opponent can do it and get an escape square for its knight. I guess the main thing is to penalise a piece a lot if it has very few safe squares to move.

[lithander]

Leorik finds it at depth 27 which took ~4 minutes to compute, single threaded.

Code: Select all

info depth 26 score cp 179 nodes 971516199 nps 7014759 time 138496 pv c4d5 d7d5 f4e3 g8e7 f1f6 e7f5 e3f4 d5d7 d4d5 d7d5 f6f5 a8g8 b2b3 d5c5 g1h1 c5d5 f5f6 d5d1 f4f1 d1f1 f6f1 e6d5 f1f6 g4g3
info depth 27 score cp 896 nodes 1753899224 nps 7136227 time 245774 pv h4h5 a8e8 f4f2 e6f5 f2f5 d7f5 f1f5 g8e7 f5f7 h7g8 f7g7 g8f8 c4d5 e7d5 g7g4 e8c8 g4g6 c8c1 g1h2 c1b1 g6h6 b1b2 h6h8 f8f7 h8h7 f7e6 h7a7

Makes a great test-position with a definitive correct answer that's hard but not impossible to find. If you've got more of those please share!

[JVMerlino]

Makes a great test-position with a definitive correct answer that's hard but not impossible to find. If you've got more of those please share!

It looks like there will be a new Myrddin within a week or so, and I'll upload my full test positions doc with the source. But for now here are a few. For the one that says "announce mate", that is the goal, although finding the correct move may be very easy. The last one is sneaky, but your engine may get it right away.
2br2k1/p1q3p1/1r2p2p/1p2Npb1/5N2/P2PQ1R1/1P3PPP/4R1K1 w - - 0 27 bm Rxg5
2br4/p1p1q1r1/1pkbB3/6NQ/2pP1P2/P3P1p1/1P6/1K2R3 w - - 0 30 bm Qf3+
3rrk2/1bnqppb1/p2p2p1/1p1P2P1/1p2PP1R/2NQBN2/1PP2K2/7R w - - bm Rh8 -- announce Mate in 16
rn3rk1/3n4/1p2pqp1/p2p2Np/3p1P2/2PQN3/PP4PP/4RRK1 w - - 0 19 bm Nxe6
4KBkr/7p/6PP/4P3/8/3P1p2/8/8 w - - bm g7

jm