Number of moves to mate as metric of playing strength

[Aleks Peshkov]

I am thinking about more sensitive frameworks to test very small engine design ideas in much less time.

1. How about using number of moves to mate in white/black game pairs as indicator of superiority?
2. And what about minimizing useless draw/draw results by creating a special test positions set consisting only of highly unbalanced starting positions?

[hgm]

For 1 it is probably better to count up to a theoretical win (as indicated by EGT). I think 2 is already done in TCEC.

[Henk]

Or maybe an idea is to make it resign early.

Or remove (or add) some pieces from initial position if it limits the game length.
Or change the chess rules. So infinite possibilities to reduce game length.

[Uri Blass]

I am thinking about more sensitive frameworks to test very small engine design ideas in much less time.

1. How about using number of moves to mate in white/black game pairs as indicator of superiority?
2. And what about minimizing useless draw/draw results by creating a special test positions set consisting only of highly unbalanced starting positions?

number of moves to mate is not a good predictor of superiority in normal chess positions

For example Stockfish does not have the knowledge that it should trade pieces with big material advantage because of some simplifications for this case and developers do not care.

Even latest stockfish version show a stupid analysis for the following position and I am sure many weak engines are going to beat stockfish in pair of games from the following position when the winner is the side who mate faster.

I guess that if you make some tournament with a lot of engines from this position when the winner is the side that mate in less moves then stockfish is not going to be in the top 100 in the following list.

https://ccrl.chessdom.com/ccrl/4040/

I think that it may be interesting to make some 9 round swiss tournament with all 498 engines in the ccrl 40/15 list from this position(every game is pair of games with white and black and the winner is the side that mate in less moves) but I do not know if somebody can get all of them to install them.


[fen]1nb1kbn1/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQ - 0 1[/fen]

Stockfish_22091707_x64_avx2:
NNUE evaluation using nn-ad9b42354671.nnue enabled
1/1 00:00 145 145k +21.96 e2-e3
2/2 00:00 539 539k +21.96 e2-e3
3/2 00:00 3k 2,620k +21.96 e2-e3
4/3 00:00 3k 2,966k +22.17 Nb1-c3 c7-c6 h2-h4
5/4 00:00 4k 3,843k +21.96 Nb1-c3 a7-a6 Ng1-f3 c7-c6
6/6 00:00 10k 4,984k +21.82 e2-e3 d7-d6
7/7 00:00 28k 7,009k +21.75 Nb1-c3 Ng8-f6 Ng1-f3 Nb8-c6 g2-g3 e7-e6
8/9 00:00 80k 9,971k +21.63 Nb1-c3 Ng8-f6 Ng1-f3 c7-c6 d2-d3 Nf6-d5 Nc3-e4 Nd5-f6
9/11 00:00 170k 11,323k +21.61 Nb1-c3 Nb8-c6 Ng1-h3 Ng8-f6 Nc3-b5 a7-a6 Nb5xc7+ Ke8-d8 Nc7-a8 Nf6-d5 e2-e4
10/11 00:00 228k 11,391k +21.61 Nb1-c3 Nb8-c6 Nc3-b5 Ke8-d8 Ng1-f3 Ng8-f6 Nf3-g5 a7-a6 Nb5-c3 Nc6-e5
11/16 00:00 560k 10,985k +21.52 Nb1-c3 Nb8-c6 Nc3-d5 Ke8-d8 Ng1-f3 e7-e6 Nd5-e3 Ng8-f6 Ne3-c4 Kd8-e8 g2-g3 Bf8-b4
12/16 00:00 764k 11,580k +21.59 Nb1-c3 Nb8-c6 a2-a3 e7-e6 Ng1-f3 Ng8-f6 e2-e4 d7-d5 e4xd5 Nf6xd5 Nc3-e4
13/18 00:00 1,556k 12,858k +21.41 Nb1-c3 Nb8-c6 d2-d4 e7-e6 a2-a3 d7-d5 f2-f3 Bf8-e7 h2-h4 Ng8-f6 e2-e3 Ke8-f8
14/22 00:00 2,074k 12,884k +21.39 Nb1-c3 Nb8-c6 d2-d4 e7-e6 a2-a3 Ng8-f6 e2-e4 d7-d5 e4-e5 Nf6-e4 Nc3-e2 f7-f6 Ng1-f3 f6xe5 d4xe5
15/24 00:00 3,020k 13,248k +21.46 e2-e3 Ng8-f6 Ng1-e2 Nb8-c6 g2-g3 b7-b6 d2-d3 e7-e6 Bf1-g2 Bf8-d6 f2-f4
16/22 00:00 5,544k 13,966k +21.58 e2-e3 Ng8-f6 Nb1-c3 Nb8-c6 a2-a3 a7-a6 d2-d4 d7-d6 Bf1-d3 Bc8-e6 Ng1-e2 Nf6-d5 Nc3xd5 Be6xd5 O-O
17/26 00:00 8,288k 14,489k +21.63 g2-g3 Ng8-f6 Bf1-g2 Nb8-c6 e2-e3 e7-e6 Ng1-e2 b7-b6 O-O Bf8-b4 a2-a3 Bb4-d6 Nb1-c3 Ke8-f8 b2-b4 Kf8-g8 f2-f4 Bc8-a6
18/21 00:00 8,686k 14,524k +21.61 g2-g3 Ng8-f6 Bf1-g2 Nb8-c6 e2-e3 e7-e6 Ng1-e2 b7-b6 O-O Bf8-d6 d2-d3 Ke8-f8 a2-a3 Bc8-a6 f2-f4 g7-g6 Nb1-c3 Ba6-c8 d3-d4 Kf8-g8
19/21 00:00 9,031k 14,520k +21.63 g2-g3 Ng8-f6 Bf1-g2 Nb8-c6 e2-e3 e7-e6 Ng1-e2 b7-b6 O-O Bf8-d6 d2-d3 Ke8-f8 Nb1-d2 Bc8-a6 a2-a3 Kf8-g8 Rf1-e1 Bd6-c5 b2-b4 Bc5-d6
20/22 00:00 9,960k 14,561k +21.67 g2-g3 Ng8-f6 Bf1-g2 Nb8-c6 e2-e3 e7-e6 Ng1-e2 b7-b6 O-O Bf8-d6 d2-d3 Ke8-f8 a2-a3 Kf8-g8 Nb1-c3 Nf6-d5 Nc3xd5 e6xd5 Bg2xd5 Nc6-e7 Bd5-g2
21/23 00:00 10,585k 14,520k +21.63 g2-g3 Ng8-f6 Bf1-g2 Nb8-c6 e2-e3 e7-e6 Ng1-e2 b7-b6 O-O Bf8-d6 d2-d3 Ke8-f8 c2-c4 a7-a5 Nb1-c3 Bc8-a6 e3-e4 g7-g6 f2-f4 Kf8-g7
22/25 00:00 12,225k 14,589k +21.61 g2-g3 Ng8-f6 Bf1-g2 Nb8-c6 e2-e3 e7-e6 Ng1-e2 b7-b6 O-O Bf8-d6 d2-d3 Ke8-f8 a2-a3 g7-g6 c2-c4 Kf8-g8 Nb1-c3 a7-a5 d3-d4 Bc8-a6 Qd1-a4 Kg8-g7 e3-e4
23/28 00:01 16,730k 14,871k +21.61 g2-g3 Ng8-f6 Bf1-g2 Nb8-c6 e2-e3 e7-e6 Ng1-e2 b7-b6 O-O Bf8-d6 d2-d3 Ke8-f8 a2-a3 g7-g6 c2-c4 Kf8-g8 Nb1-c3 a7-a5 d3-d4 Bc8-a6 Qd1-a4 Kg8-g7 e3-e4
24/29 00:01 19,181k 14,974k +21.61 g2-g3 Nb8-c6 Bf1-g2 e7-e6 e2-e3 b7-b6 d2-d3 Ng8-f6 a2-a3 Bf8-d6 Ng1-e2 Ke8-f8 O-O a7-a5 c2-c4 g7-g6 Nb1-c3 Bc8-a6 e3-e4 Kf8-g7 f2-f4 Ba6-b7 e4-e5
25/30 00:01 29,471k 15,113k +21.61 g2-g3 Nb8-c6 Bf1-g2 e7-e6 e2-e3 b7-b6 d2-d3 Ng8-e7 Nb1-d2 g7-g6 Ng1-e2 Bf8-g7 O-O Ke8-f8 a2-a3 Ne7-f5 d3-d4 a7-a5 c2-c4 Kf8-g8 Qd1-a4 Bc8-a6 b2-b4 Nf5-d6 b4-b5
26/32 00:02 36,241k 15,215k +21.61 g2-g3 Nb8-c6 e2-e3 e7-e6 Bf1-g2 b7-b6 d2-d3 Ng8-e7 Nb1-d2 g7-g6 Ng1-e2 Bf8-g7 O-O Ne7-f5 c2-c3 Nc6-e5 Qd1-c2 Bc8-a6 c3-c4 Ne5-c6 Bg2-h3 Ba6-b7 d3-d4 Ke8-f8 Qc2-a4 Nf5-d6 d4-d5
27/34 00:02 45,468k 15,232k +21.61 g2-g3 Nb8-c6 e2-e3 e7-e6 Bf1-g2 b7-b6 d2-d3 Ng8-e7 Ng1-e2 g7-g6 O-O Bf8-g7 Nb1-d2 Ke8-f8 c2-c3 Bc8-a6 c3-c4 Ba6-b7 Qd1-a4 a7-a5 Qa4-b5 Kf8-g8 f2-f4 Ne7-f5 Nd2-e4 Nf5-d6 Ne4xd6 c7xd6
28/36 00:04 62,161k 15,273k +21.53 g2-g3 Nb8-c6 d2-d3 e7-e6 Bf1-g2 b7-b6 e2-e3 Ng8-e7 Ng1-e2 g7-g6 O-O Bc8-b7 f2-f4 Bf8-g7 c2-c3 Bb7-a6 Nb1-a3 Ke8-f8 Ra1-b1 f7-f5 b2-b4 Nc6-d8 Bg2-h3 Nd8-c6 b4-b5
29/35 00:05 79,784k 15,311k +21.61 g2-g3 Nb8-c6 e2-e3 e7-e6 Bf1-g2 b7-b6 Ng1-e2 Ng8-e7 O-O g7-g6 d2-d3 Ke8-d8 f2-f4 Bc8-a6 c2-c4 Kd8-c8 Nb1-d2 Ne7-f5 Nd2-e4 Bf8-e7 a2-a3 Ba6-b7 g3-g4 Nf5-h4 Bg2-h3 Kc8-b8 Ne4-d2 Be7-c5 d3-d4 Bc5-d6
30/34 00:07 107,886k 15,296k +21.52 g2-g3 Nb8-c6 e2-e3 e7-e6 Bf1-g2 b7-b6 Ng1-e2 Ng8-e7 O-O g7-g6 d2-d3 Bc8-a6 Nb1-d2 Ne7-f5 Nd2-f3 Bf8-g7 c2-c3 Nf5-d6 Nf3-e1 Ba6-b7 Qd1-a4 a7-a5 a2-a3 Ke8-f8 f2-f4 Kf8-g8 Ne1-f3 Bb7-a6 Qa4-c2 Nd6-b7 b2-b4 Nb7-d6

[Aleks Peshkov]

Or maybe an idea is to make it resign early.

Or remove (or add) some pieces from initial position if it limits the game length.
Or change the chess rules. So infinite possibilities to reduce game length.

I am not trying to limit single test game length. I am trying to reduce number of test games needed to proof or disproof a design hypothesis.

[Aleks Peshkov]

number of moves to mate is not a good predictor of superiority in normal chess positions

For example Stockfish does not have the knowledge that it should trade pieces with big material advantage because of some simplifications for this case and developers do not care.

Even latest stockfish version show a stupid analysis for the following position and I am sure many weak engines are going to beat stockfish in pair of games from the following position when the winner is the side who mate faster.

I guess that if you make some tournament with a lot of engines from this position when the winner is the side that mate in less moves then stockfish is not going to be in the top 100 in the following list.

Even if Stockfish does not understand the need of fast win it will fight to avoid fast lose from the weaker side. So in two games mini-match it does not matter how fast theoretical shortest mate is but only who wins faster (or avoid lose longer).

And I am going to incrementally test engine play against earlier version of itself while the engine in its rudiment state.

[Uri Blass]

number of moves to mate is not a good predictor of superiority in normal chess positions

For example Stockfish does not have the knowledge that it should trade pieces with big material advantage because of some simplifications for this case and developers do not care.

Even latest stockfish version show a stupid analysis for the following position and I am sure many weak engines are going to beat stockfish in pair of games from the following position when the winner is the side who mate faster.

I guess that if you make some tournament with a lot of engines from this position when the winner is the side that mate in less moves then stockfish is not going to be in the top 100 in the following list.

Even if Stockfish does not understand the need of fast win it will fight to avoid fast lose from the weaker side. So in two games mini-match it does not matter how fast theoretical shortest mate is but only who wins faster (or avoid lose longer).

And I am going to incrementally test engine play against earlier version of itself while the engine in its rudiment state.

Stockfish has bad evaluation and does not understand to avoid losing fast.

[pgn][Event "banksia game"]
[Site "*"]
[Date "2022.09.17"]
[Round "*"]
[White "Wasp 4.50"]
[Black "Stockfish 170922"]
[Result "1-0"]
[TimeControl "60+1"]
[Time "19:26:17"]
[Termination "mate"]
[FEN "1nb1kbn1/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQ - 0 1"]
[SetUp "1"]

1.Nf3 {+23.17/19 2767 30976627} b6 {+21.37/24 8325 126940981} 2.d4 {+23.57/19 2766 30797243} e6 {+21.21/26 1152 15887036}
3.Nc3 {+23.97/20 2072 23206091} Bb4 {+20.97/25 1763 24189764} 4.Bf4 {+24.82/20 2227 25412451} Bxc3+ {+20.97/29 4622 66082811}
5.bxc3 {+25.89/22 2106 25171144} d6 {+21.18/27 5332 74738429} 6.Qd3 {+26.69/23 2125 24021462} Nf6 {+22.10/25 12244 161066122}
7.e4 {+27.39/22 2145 24101987} Ba6 {+22.32/24 11615 164416375} 8.c4 {+27.51/22 2170 25813610} Bb7 {+22.76/22 10842 157884871}
9.e5 {+30.33/19 2190 25447279} Nh5 {+23.02/21 9659 152228084} 10.Qxh7 {+34.88/20 2215 28363820} Nxf4 {+24.73/19 2732 45443148}
11.Qg8+ {+37.13/20 2179 31518713} Kd7 {+35.61/16 1010 15786586} 12.Qxf7+ {+38.67/18 2138 32054849} Kd8 {+41.37/17 974 14850809}
13.Qxf4 {+42.05/18 2102 31925404} Nd7 {+25.30/15 1359 21910045} 14.Qh4+ {+49.53/19 6491 90443140} Ke8 {M-15/24 669 7450965}
15.Qh8+ {M+11/15 2528 33218446} Ke7 {M-12/33 749 8744890} 16.Ng5 {M+10/15 1840 25262166} Bc6 {M-11/34 300 3310551}
17.Qxg7+ {M+9/15 1816 23531922} Kd8 {M-10/38 364 3956548} 18.Qg8+ {M+8/15 960 12099531} Ke7 {M-7/39 889 11379252}
19.d5 {M+7/13 81 628627} Ba4 {M-7/45 436 5035050} 20.Qxe6+ {M+6/11 112 1174754} Kd8 {M-5/72 478 5957791}
21.c5 {M+5/9 24 150004} Nxe5 {M-4/128 630 17761421} 22.c6 {M+4/7 14 32842} Bxc6 {M-3/245 62 2342271}
23.dxc6 {M+3/5 13 935} b5 {M-2/245 10 126038} 24.Nf7+ {M+2/3 12 329} Nxf7 {M-1/245 10 3437}
25.Qd7# {M+1/1 13 82} 1-0

[/pgn]

Old Wasp withn no NN survive against itself for more moves

[pgn][Event "banksia game"]
[Site "*"]
[Date "2022.09.17"]
[Round "*"]
[White "Wasp 4.50"]
[Black "Wasp 4.50"]
[Result "1-0"]
[TimeControl "60+1"]
[Time "19:29:15"]
[Termination "mate"]
[FEN "1nb1kbn1/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQ - 0 1"]
[SetUp "1"]

1.e4 {+23.32/18 2767 29201195} d6 {+23.35/19 2768 29329856} 2.d4 {+23.42/19 2071 21952522} a6 {+23.43/20 2780 29636181}
3.Nf3 {+23.55/20 2090 22309683} e6 {+23.56/19 2768 29511376} 4.c4 {+23.65/18 2109 22628206} h6 {+23.79/19 2769 29816613}
5.Nc3 {+23.86/20 2126 22480297} Ne7 {+24.17/20 2768 29553662} 6.Be3 {+24.08/21 2146 20785159} Nd7 {+24.18/19 2771 26936641}
7.Bd3 {+24.32/19 2167 20978889} c6 {+24.39/20 2782 26803320} 8.O-O {+24.73/22 2189 21138345} c5 {+25.39/20 2770 26730310}
9.dxc5 {+25.89/21 2215 22239693} Nc6 {+26.48/21 2772 26861176} 10.cxd6 {+27.66/21 2236 22777022} Bxd6 {+27.66/20 2773 28500128}
11.Bc2 {+28.73/20 2197 23896825} Be7 {+28.92/20 2699 29175009} 12.Ba4 {+29.59/20 2159 23628031} f6 {+31.32/21 3677 41413672}
13.Nd4 {+32.71/21 2123 22715224} Nde5 {+32.15/20 1878 20628754} 14.Nxc6 {+34.13/20 2788 31431889} bxc6 {+35.51/21 1850 21877171}
15.f4 {+37.64/21 2031 24731408} Nf7 {+38.45/20 2751 34432886} 16.Bxc6+ {+40.43/22 1997 24800974} Kf8 {+39.37/21 1766 21578285}
17.Bd7 {+40.09/21 1965 24446929} Bxd7 {+45.76/20 8594 111979965} 18.Qxd7 {+48.85/20 5954 79027140} Nd8 {+48.72/17 2208 27170448}
19.Bb6 {M+11/16 1773 21084675} h5 {M-10/15 2323 28241674} 20.Bxd8 {M+8/15 593 6533990} Bc5+ {M-8/17 1408 19369635}
21.Kh1 {M+7/13 13 3368} h4 {M-7/13 73 676285} 22.Qxe6 {M+6/11 104 925666} h3 {M-6/11 24 132718}
23.Rad1 {M+5/9 15 10250} hxg2+ {M-5/9 15 4016} 24.Kxg2 {M+4/7 13 1423} Bb4 {M-4/7 14 4256}
25.Rd7 {M+3/5 13 3632} Be7 {M-3/5 13 490} 26.Bxe7+ {M+2/3 13 393} Ke8 {M-2/3 13 7}
27.Ba3# {M+1/1 15 106} 1-0

[/pgn]

Finally old Wasp survive against stockfish for significantly more moves

[pgn][Event "banksia game"]
[Site "*"]
[Date "2022.09.17"]
[Round "*"]
[White "Stockfish 170922"]
[Black "Wasp 4.50"]
[Result "1-0"]
[TimeControl "60+1"]
[Time "19:33:27"]
[Termination "mate"]
[FEN "1nb1kbn1/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQ - 0 1"]
[SetUp "1"]

1.g3 {+21.61/25 8901 7561627} Nf6 {+22.93/19 2939 23655176} 2.Bg2 {+21.53/25 1626 13854422} e6 {+22.96/18 2764 26297954}
3.e3 {+21.61/28 3471 43944111} d5 {+22.89/17 2764 27265066} 4.Ne2 {+21.61/28 1250 15411713} a6 {+22.92/17 2765 26805382}
5.O-O {+21.58/26 5033 67266912} b5 {+23.19/17 2763 25046030} 6.d3 {+21.70/25 2874 35027001} Nbd7 {+23.39/19 2764 27165346}
7.Nd2 {+21.67/30 2941 37368007} Bb7 {+23.62/18 2766 28176168} 8.c3 {+21.75/25 2716 33677816} h6 {+23.53/18 2774 27256537}
9.a4 {+21.71/25 3186 39652957} b4 {+24.67/21 2768 27697586} 10.Qc2 {+21.87/23 1911 23769938} c6 {+24.96/21 2768 28897845}
11.c4 {+22.20/21 1791 22381509} c5 {+24.94/19 2693 27346397} 12.cxd5 {+22.38/24 1847 23349078} exd5 {+26.39/19 2619 26962763}
13.Nb3 {+22.70/20 1914 24648898} Bd6 {+27.19/21 2549 26163653} 14.Bd2 {+22.96/22 1952 25627135} Ke7 {+26.70/22 1858 19700726}
15.d4 {+23.77/23 2109 27757165} c4 {+28.32/22 1830 20084594} 16.Na5 {+23.85/22 2036 26214119} Ba8 {+29.22/22 1804 19527866}
17.Nxc4 {+24.46/22 3378 44439432} dxc4 {+29.90/23 2050 22081717} 18.Bxa8 {+24.52/22 2251 29289901} Nb6 {+30.33/22 1744 18454509}
19.Bc6 {+24.52/19 2487 33600821} a5 {+32.04/19 2299 24648927} 20.Qf5 {+24.80/20 2312 31742603} c3 {+33.40/19 2242 24558571}
21.bxc3 {+25.87/19 2718 38323589} Nc4 {+35.40/18 2189 24513228} 22.Bc1 {+26.58/18 2097 31210774} g6 {+33.86/18 2144 22905436}
23.Qb5 {+27.22/18 2149 32575544} Na3 {+36.64/18 2086 22229849} 24.Bxa3 {+29.89/17 2323 33909492} Kf8 {+44.88/18 3454 48804413}
25.cxb4 {+33.24/19 3764 51487106} axb4 {+48.52/18 1931 23616194} 26.Qxb4 {+34.86/15 2383 33508077} Bxb4 {M-13/20 4046 47421808}
27.Bxb4+ {+36.60/13 2817 38974368} Kg7 {M-12/20 1309 14423121} 28.a5 {+37.05/16 2380 32441926} Nh7 {M-11/18 1737 19871369}
29.Bc3 {+39.38/16 2497 34581519} h5 {M-11/16 1703 19418396} 30.d5+ {+41.37/15 2596 35555058} f6 {M-10/17 1247 14057756}
31.a6 {+62.55/25 2386 25101178} Ng5 {M-9/17 585 6579127} 32.a7 {M+9/37 1113 13854741} Nf3+ {M-8/15 17 82294}
33.Kg2 {M+8/42 1147 16907640} Ne5 {M-8/15 655 7729924} 34.Bxe5 {M+7/45 1102 17825204} h4 {M-7/13 127 1368918}
35.a8=Q {M+6/43 1265 21191751} h3+ {M-6/11 13 9394} 36.Kf3 {M+5/108 999 29523957} g5 {M-5/9 13 7373}
37.Ra7+ {M+4/245 657 23956574} Kg6 {M-4/7 15 2060} 38.Qg8+ {M+3/245 14 401143} Kf5 {M-3/5 15 474}
39.Nd4+ {M+2/245 12 101753} Kxe5 {M-2/3 11 8} 40.Qe6# {M+1/245 12 87475} 1-0

[/pgn]