Comparison wanted: mate in 7 test

[Chessnut1071]

I'm testing my new engine which is based on mailbox ideas; however, it's not really a mailbox in a formal sense. It's 21x faster than my old inefficient engine, but, I don't have any idea how it compares with a well structured mailbox or bitboard program. So, I tested it on the FEN below with the following statistics:

FEN[201] = "5k2/ppp2r1p/2p2ppP/8/2Q5/2P1bN2/PP4P1/1K1R4 w - - 0 1 "; // Unknown author 7-move mate [Qe4]

engine calls: 847,550,776
nodes visited: 34,141233,773
time: 11,327.6 seconds

methods: alpha/beta, history by ply, Zobirst hash, optimized evaluation function: history, capture, check, slider piece, enemy king moves[feedback[], and pawn promotion.

computer chip Intel I5, 2.5 GHz, 16GB ram

Need somebody to compare with.

[amanjpro]

I'm testing my new engine which is based on mailbox ideas; however, it's not really a mailbox in a formal sense. It's 21x faster than my old inefficient engine, but, I don't have any idea how it compares with a well structured mailbox or bitboard program. So, I tested it on the FEN below with the following statistics:

FEN[201] = "5k2/ppp2r1p/2p2ppP/8/2Q5/2P1bN2/PP4P1/1K1R4 w - - 0 1 "; // Unknown author 7-move mate [Qe4]

engine calls: 847,550,776
nodes visited: 34,141233,773
time: 11,327.6 seconds

methods: alpha/beta, history by ply, Zobirst hash, optimized evaluation function: history, capture, check, slider piece, enemy king moves[feedback[], and pawn promotion.

computer chip Intel I5, 2.5 GHz, 16GB ram

Need somebody to compare with.

Zahak is not really much better, with normal search, 2.3GHz of CPU, 1GB Hash

Code: Select all

info depth 35 seldepth 23 hashfull 391 tbhits 0 nodes 82803802 nps 937253 score mate +7 time 88347 pv c4e4 e3h6 d1d8 f8g7 f3d4 h6f4 d4e6 g7h6 d8d3 h6h5 d3h3 h5g4

[Chessnut1071]

I'm testing my new engine which is based on mailbox ideas; however, it's not really a mailbox in a formal sense. It's 21x faster than my old inefficient engine, but, I don't have any idea how it compares with a well structured mailbox or bitboard program. So, I tested it on the FEN below with the following statistics:

FEN[201] = "5k2/ppp2r1p/2p2ppP/8/2Q5/2P1bN2/PP4P1/1K1R4 w - - 0 1 "; // Unknown author 7-move mate [Qe4]

engine calls: 847,550,776
nodes visited: 34,141233,773
time: 11,327.6 seconds

methods: alpha/beta, history by ply, Zobirst hash, optimized evaluation function: history, capture, check, slider piece, enemy king moves[feedback[], and pawn promotion.

computer chip Intel I5, 2.5 GHz, 16GB ram

Need somebody to compare with.

Zahak is not really much better, with normal search, 2.3GHz of CPU, 1GB Hash

Code: Select all

info depth 35 seldepth 23 hashfull 391 tbhits 0 nodes 82803802 nps 937253 score mate +7 time 88347 pv c4e4 e3h6 d1d8 f8g7 f3d4 h6f4 d4e6 g7h6 d8d3 h6h5 d3h3 h5g4

do you know if that engine is based on bitboards or mailboxes?

[amanjpro]

I'm testing my new engine which is based on mailbox ideas; however, it's not really a mailbox in a formal sense. It's 21x faster than my old inefficient engine, but, I don't have any idea how it compares with a well structured mailbox or bitboard program. So, I tested it on the FEN below with the following statistics:

FEN[201] = "5k2/ppp2r1p/2p2ppP/8/2Q5/2P1bN2/PP4P1/1K1R4 w - - 0 1 "; // Unknown author 7-move mate [Qe4]

engine calls: 847,550,776
nodes visited: 34,141233,773
time: 11,327.6 seconds

methods: alpha/beta, history by ply, Zobirst hash, optimized evaluation function: history, capture, check, slider piece, enemy king moves[feedback[], and pawn promotion.

computer chip Intel I5, 2.5 GHz, 16GB ram

Need somebody to compare with.

Zahak is not really much better, with normal search, 2.3GHz of CPU, 1GB Hash

Code: Select all

info depth 35 seldepth 23 hashfull 391 tbhits 0 nodes 82803802 nps 937253 score mate +7 time 88347 pv c4e4 e3h6 d1d8 f8g7 f3d4 h6f4 d4e6 g7h6 d8d3 h6h5 d3h3 h5g4

do you know if that engine is based on bitboards or mailboxes?

It is bitboards, but that hardly matters in this setting I believe

[gaard]

I'm testing my new engine which is based on mailbox ideas; however, it's not really a mailbox in a formal sense. It's 21x faster than my old inefficient engine, but, I don't have any idea how it compares with a well structured mailbox or bitboard program. So, I tested it on the FEN below with the following statistics:

FEN[201] = "5k2/ppp2r1p/2p2ppP/8/2Q5/2P1bN2/PP4P1/1K1R4 w - - 0 1 "; // Unknown author 7-move mate [Qe4]

engine calls: 847,550,776
nodes visited: 34,141233,773
time: 11,327.6 seconds

methods: alpha/beta, history by ply, Zobirst hash, optimized evaluation function: history, capture, check, slider piece, enemy king moves[feedback[], and pawn promotion.

computer chip Intel I5, 2.5 GHz, 16GB ram

Need somebody to compare with.

Code: Select all

2021-11-26 00:38:31.239<--1:info depth 21 seldepth 39 score cp 1679 time 1422 nodes 2057548 nps 1446939 hashfull 13 pv d1d8 f8e7 d8d7 e7d7 c4f7 d7d6 f7f6 d6c5 f6e5 c5b6 e5e3 b6a6 e3e5 c6c5 e5c7 c5c4 c7c4 a6b6 f3d4 a7a6 c4g8 b6a7 g8h7 g6g5 h7g6 g5g4 g6g4 b7b5
2021-11-26 00:38:39.731<--1:info depth 22 seldepth 40 score cp 2005 time 9915 nodes 12509684 nps 1261692 hashfull 109 pv d1d8 f8e7 d8c8 e7d6 c4f7 d6c5 c8c7 c5b6 c7b7 b6a6 f7d7 e3b6 d7c6 f6f5 b7h7 f5f4 c6g6 a6b5 g6f5 b6c5 h7a7 b5b6 h6h7 c5e3
2021-11-26 00:38:51.038<--1:info depth 23 seldepth 40 score cp 2195 time 21222 nodes 26213687 nps 1235212 hashfull 240 pv d1d8 f8e7 d8c8 e7d6 c4f7 d6c5 f7c7 c5d5 c7b7 d5e4 c8e8 e4f5 e8e3 f5f4 b7a7 f6f5 e3e6 f4g3 a7h7 g3g2 f3d4 g2f1 e6c6 f5f4 c6g6
2021-11-26 00:40:12.896<--1:info depth 24 seldepth 45 score mate 14 time 103080 nodes 142268666 nps 1380177 hashfull 384 pv d1d8 f8e7 d8c8 e7d6 c4f7 d6c5 f7c7 c5d5 c7f7 d5d6 f7b7 d6e6 c8c6 e6f5 b7d7 f5f4 c6f6 f4g3 d7h3 g3f2 f3d4 e3f4 f6f4 f2e1 h3e3 e1d1 e3e2
2021-11-26 00:40:13.179<--1:info depth 25 seldepth 45 score mate 14 time 103363 nodes 142699047 nps 1380562 hashfull 384 pv d1d8 f8e7 d8c8 e7d6 c4f7 d6c5 f7c7 c5d5 c7f7 d5d6 f7b7 d6e6 c8c6 e6f5 b7d7 f5f4 c6f6 f4g3 d7h3 g3f2 f3d4 e3f4 f6f4 f2e1 h3e3 e1d1 e3e2
2021-11-26 00:40:16.419<--1:info depth 26 seldepth 45 score mate 9 time 106603 nodes 147825840 nps 1386694 hashfull 388 pv d1d8 f8e7 d8c8 a7a5 c8c7 e7d6 c7f7 e3f2 f7b7 a5a4 c4d3 d6c5 b2b3 a4b3 a2b3 f2g1 b3b4
2021-11-26 00:40:16.886<--1:info depth 27 seldepth 45 score mate 9 time 107070 nodes 148592003 nps 1387802 hashfull 387 pv d1d8 f8e7 d8c8 a7a5 c8c7 e7d6 c7f7 e3f2 f7b7 a5a4 c4d3 d6c5 b2b3 a4b3 a2b3 f2g1 b3b4
2021-11-26 00:40:17.387<--1:info depth 28 seldepth 45 score mate 9 time 107570 nodes 149431799 nps 1389158 hashfull 387 pv d1d8 f8e7 d8c8 a7a5 c8c7 e7d6 c7f7 e3f2 f7b7 a5a4 c4d3 d6c5 b2b3 a4b3 a2b3 f2g1 b3b4
2021-11-26 00:40:18.318<--1:info depth 29 seldepth 45 score mate 9 time 108502 nodes 150977141 nps 1391468 hashfull 384 pv d1d8 f8e7 d8c8 a7a5 c8c7 e7d6 c7f7 e3f2 f7b7 a5a4 c4d3 d6c5 b2b3 a4b3 a2b3 f2g1 b3b4
2021-11-26 00:40:19.669<--1:info depth 30 seldepth 45 score mate 9 time 109853 nodes 153265986 nps 1395191 hashfull 383 pv d1d8 f8e7 d8c8 a7a5 c8c7 e7d6 c7f7 e3f2 f7b7 a5a4 c4d3 d6c5 b2b3 a4b3 a2b3 f2g1 b3b4
2021-11-26 00:40:25.904<--1:info depth 31 seldepth 45 score mate 7 time 116087 nodes 162948550 nps 1403676 hashfull 381 pv c4e4 e3h6 d1d8 f8g7 f3d4 f7f8 e4e7 g7g8 d8d7 f8f7 e7f7 g8h8 f7h7
2021-11-26 00:40:26.128<--1:info depth 31 seldepth 45 score mate 7 time 116312 nodes 163354599 nps 1404451 hashfull 381 pv c4e4 e3h6 d1d8 f8g7 f3d4 f7f8 e4e7 g7g8 d8d7 f8f7 e7f7 g8h8 f7h7
2021-11-26 00:40:26.810<--1:info depth 32 seldepth 45 score mate 7 time 116994 nodes 164510971 nps 1406148 hashfull 382 pv c4e4 e3h6 d1d8 f8g7 f3d4 f7f8 e4e7 g7g8 d8d7 f8f7 e7f7 g8h8 f7h7
2021-11-26 00:40:27.901<--1:info depth 33 seldepth 45 score mate 7 time 118084 nodes 166466820 nps 1409732 hashfull 380 pv c4e4 e3h6 d1d8 f8g7 f3d4 f7f8 e4e7 g7g8 d8d7 f8f7 e7f7 g8h8 f7h7
2021-11-26 00:40:28.816<--1:info depth 34 seldepth 45 score mate 7 time 118999 nodes 168120377 nps 1412788 hashfull 379 pv c4e4 e3h6 d1d8 f8g7 f3d4 f7f8 e4e7 g7g8 d8d7 f8f7 e7f7 g8h8 f7h7
2021-11-26 00:40:28.976-->1:stop
2021-11-26 00:40:28.980<--1:bestmove c4e4

Keele finds it in about 2 minutes. Keele is a relatively simple mailbox engine rated a little under 2600. SEE, check extensions, NMP, razoring, transposition table, PVS with aspiration windows, LMR, eval pruning, history scoring/reductions, and killer moves.

[MartinBryant]

FEN[201] = "5k2/ppp2r1p/2p2ppP/8/2Q5/2P1bN2/PP4P1/1K1R4 w - - 0 1 "; // Unknown author 7-move mate [Qe4]

Colossus 2021b finds a mate in 14 at depth 11 after just over 1s...
info depth 11 time 1048 nodes 2496618 score mate 14 pv d1d8 f8e7 d8c8 a7a5 c4e4 e7d6 e4d3 e3d4 d3d4 d6e7 d4e4 e7d7 e4e8 d7d6 c8d8 d6c5 e8e3 c5b5 f3d4 b5b6 d4e6 b6a6 d8a8 a6b5 e3c5 b5a4 a8a5

...and refines it down to mate in 7 at depth 15 after just under 5s...
info depth 15 time 4970 nodes 10369677 score mate 7 pv c4e4 e3h6 d1d8 f8g7 f3d4 f7f8 e4e7 g7g8 d8d7 f8a8 e7h7 g8f8 h7f7

AMD Ryzen 5 3600 3.59GHz
32MB hash
Bitboard engine with the usual stuff.

[hgm]

Fairy-Max 5.0b requires 436 sec on a 3.2GHz i7:

Code: Select all

 17	    #7 	682.8M	7:16.30	1. Qe4 Bxh6 2. Rd8 Kg7 3. Nd4 Bg5 4. Ne6 Kh6 5. Qg4 c5 6. Qh3 Bh4 7. Qxh4 
 17	    #8 	612.2M	6:32.54	1. Rd8 Ke7 2. Rc8 a5 3. Rxc7 Kd6 4. Qxf7 Kc5 5. Rxb7 a4 6. Ne1 Bxh6 7. Nd3 Kd6 8. Qd7 
 16	    #8 	290.6M	3:08.37	1. Rd8 Ke7 2. Rc8 a5 3. Rxc7 Kd6 4. Qxf7 Kc5 5. Rxb7 Kd6 6. b3 Bxh6 7. Qe7 Kd5 8. c4 
 15	    #9 	128.1M	1:24.22	1. Rd8 Ke7 2. Rc8 a5 3. Qe4 Kd6 4. Qd3 Kc5 5. Qxe3 Kb5 6. c4 Kb4 7. Nd2 a4 8. a3 Ka5 9. Ra8 
 14	    #9 	56.6M  	0:37.81	1. Rd8 Ke7 2. Rc8 a5 3. Qe4 Kd6 4. Qd3 Kc5 5. Qxe3 Kb5 6. c4 Kb4 7. Nd2 a4 8. a3 Ka5 9. Ra8 
 13	  #10 	33.6M  	0:22.57	1. Rd8 Ke7 2. Rc8 a5 3. Qe4 Kd6 4. Qd3 Kc5 5. Qxe3 Kb5 6. c4 Kb4 7. a3 Kxc4 8. Qd4 Kb5 9. a4 Ka6 
 12	+15.37 	11.4M  	0:07.58	1. Rd8 Ke7 2. Rc8 Bf2 3. Rxc7 Kd6 4. Rxf7 a5 5. Rxf6 Kc7 6. Qf7 Kb6 7. Qxh7 g5 8. Qe4 Bc5 9. Nxg5 
 11	+14.65 	3.52M  	0:02.34	1. Rd8 Ke7 2. Rc8 f5 3. Rxc7 Kd6 4. Rxf7 f4 5. Rf6 Kc7 6. Qf7 Kb6 7. Qxh7 a5 8. Qxg6 
 10	+12.79 	1.12M  	0:00.74	1. Rd8 Ke7 2. Rd7 Kxd7 3. Qxf7 Kd8 4. Qxf6 Kc8 5. Qe6 Kd8 6. Qxe3 c5 7. Qg5 Kc8 8. Qxc5 a6 
  9	+12.35 	426527	0:00.31	1. Rd8 Ke7 2. Rd7 Kxd7 3. Qxf7 Kd8 4. Qxf6 Kc8 5. Qe6 Kd8 
  8	+11.86 	278396	0:00.20	1. Rd8 Ke7 2. Rd7 Kxd7 3. Qxf7 Kd8 4. Qxf6 Ke8 5. Qe6 Kf8 6. Qxe3 a5 7. a4 
  7	+10.77 	155942	0:00.12	1. Rd8 Ke7 2. Qd3 f5 3. Rd7 Ke6 4. Rxf7 Kxf7 5. Qxe3 
  6	+9.63 	27272  	0:00.03	1. Rd8 Ke7 2. Rd7 Kxd7 3. Qxf7 Kd6 4. Qxf6 Kd7 5. Qf7 Kd6 6. Qxh7 
  5	+8.68 	12053  	0:00.01	1. Rd8 Ke7 2. Rd7 Kxd7 3. Qxf7 Kd6 4. Qxf6 Kd5 5. Qe5 
  4	+8.51 	4490    	0:00.00	1. Rd8 Ke7 2. Rd7 Kxd7 3. Qxf7 Kd6 4. Qxh7 
  3	+7.62 	3298    	0:00.00	1. Rd8 Ke7 2. Rd7 Kxd7 3. Qxf7 
  3	+7.48 	2411    	0:00.00	1. Qb4 c5 2. Qxb7 Bxh6 
  2	+7.48 	297      	0:00.00	1. Qb4 Re7 2. Qxb7 
  2	+7.17 	159      	0:00.00	1. Qh4 a5 
  1	+7.30 	47        	0:00.00	1. Qh4 

[dangi12012]

I'm testing my new engine which is based on mailbox ideas; however, it's not really a mailbox in a formal sense. It's 21x faster than my old inefficient engine, but, I don't have any idea how it compares with a well structured mailbox or bitboard program. So, I tested it on the FEN below with the following statistics:

FEN[201] = "5k2/ppp2r1p/2p2ppP/8/2Q5/2P1bN2/PP4P1/1K1R4 w - - 0 1 "; // Unknown author 7-move mate [Qe4]

engine calls: 847,550,776
nodes visited: 34,141233,773
time: 11,327.6 seconds

methods: alpha/beta, history by ply, Zobirst hash, optimized evaluation function: history, capture, check, slider piece, enemy king moves[feedback[], and pawn promotion.

computer chip Intel I5, 2.5 GHz, 16GB ram

Need somebody to compare with.

You know that this is totally useless?
One engine can perform better than others - but this is just random noise.

You would need thousands of "mate in X" positions to actually get engine or algorithmic performance test.

[Chessnut1071]

I'm testing my new engine which is based on mailbox ideas; however, it's not really a mailbox in a formal sense. It's 21x faster than my old inefficient engine, but, I don't have any idea how it compares with a well structured mailbox or bitboard program. So, I tested it on the FEN below with the following statistics:

FEN[201] = "5k2/ppp2r1p/2p2ppP/8/2Q5/2P1bN2/PP4P1/1K1R4 w - - 0 1 "; // Unknown author 7-move mate [Qe4]

engine calls: 847,550,776
nodes visited: 34,141233,773
time: 11,327.6 seconds

methods: alpha/beta, history by ply, Zobirst hash, optimized evaluation function: history, capture, check, slider piece, enemy king moves[feedback[], and pawn promotion.

computer chip Intel I5, 2.5 GHz, 16GB ram

Need somebody to compare with.

You know that this is totally useless?
One engine can perform better than others - but this is just random noise.

You would need thousands of "mate in X" positions to actually get engine performance test.

I strongly disagree, it's not as useless as you think. When you have no information, any information is better than none. Also, deep mate in x puzzles tests trillions of nodes and deep ply searches. Any efficient engine will act proportionally to 10s of 1,000s of game iterations without the time overhead. I agree it's not ELO, but, the most efficient engines with respect to ELO will also tend to be the most efficient at mate in x puzzles. If somebody has access to the major engines with reported ELO scores we can make a direct correlation and see who's right.

[dangi12012]

I'm testing my new engine which is based on mailbox ideas; however, it's not really a mailbox in a formal sense. It's 21x faster than my old inefficient engine, but, I don't have any idea how it compares with a well structured mailbox or bitboard program. So, I tested it on the FEN below with the following statistics:

FEN[201] = "5k2/ppp2r1p/2p2ppP/8/2Q5/2P1bN2/PP4P1/1K1R4 w - - 0 1 "; // Unknown author 7-move mate [Qe4]

engine calls: 847,550,776
nodes visited: 34,141233,773
time: 11,327.6 seconds

methods: alpha/beta, history by ply, Zobirst hash, optimized evaluation function: history, capture, check, slider piece, enemy king moves[feedback[], and pawn promotion.

computer chip Intel I5, 2.5 GHz, 16GB ram

Need somebody to compare with.

You know that this is totally useless?
One engine can perform better than others - but this is just random noise.

You would need thousands of "mate in X" positions to actually get engine performance test.

I strongly disagree, it's not as useless as you think. When you have no information, any information is better than none. Also, deep mate in x puzzles tests trillions of nodes and deep ply searches. Any efficient engine will act proportionally to 10s of 1,000s of game iterations without the time overhead. I agree it's not ELO, but, the most efficient engines with respect to ELO will also tend to be the most efficient at mate in x puzzles. If somebody has access to the major engines with reported ELO scores we can make a direct correlation and see who's right.

Engine A finds this by pure chance after a few million nodes. Engine B finds this with its algo a lot later but has a much higher elo.
Thats why you need a lot of positions to get rid of NOISE.

Anyways for SF14.1 on a Ryzen 5950X the output is this:
Same machine after reboot (to clear the TB cache)

Code: Select all

Run A)
62/16	00:04	 165.515k	39.474k	+M7	Qc4-e4 Be3xh6 Rd1-d8+ Kf8-g7 Nf3-d4 Bh6-f4 Nd4-e6+ Kg7-h6 Rd8-d3 Kh6-h5 Rd3-h3+ Kh5-g4 Qe4xf4+
Run B)
33/16	00:00	 9.163k	57.271k	+M7	Qc4-e4 Be3xh6 Rd1-d8+ Kf8-g7 Nf3-d4 Rf7-f8 Qe4-e7+ Kg7-g8 Rd8-d7 Rf8-d8 Rd7xd8+ Bh6-f8 Qe7xf8+

Soooo. I just proved my point. Same engine same hardware - looks as if Run B is a much better engine. But in reallity its the same.
So this test run as a single test is not only saying nothing - its heavily misleading.

Better have 1000 Mate in X tests to get proper results.