A look back at the Engine Crackers

Vinvin · Post by **Vinvin** » Thu Mar 25, 2021 2:25 am

In June 2020, Glarean posted 18 "Engine Crackers" positions.
http://talkchess.com/forum3/viewtopic.p ... 25#p846325
All the 8 positions with variations here : https://glarean-magazin.ch/2020/06/12/c ... s-puzzles/

I let run the 5 solvers + SF 13 for 15 minutes (900 seconds) per position. CPU : old i7-6*4GHz, 1 thread used.

On the 18 problems :
Crystal 3.1 solved 11
Bluefish-v13 solved 17 <- best score
Black-Diamond-v13 solved 10
SugaR.AI.ICCF.1.00 solved 14
ShashChess15.1 solved 12
Stockfish-13 solved 13

Table (results are in second unit).

Code: Select all

18 Crackers    Bluefish-13 Black-D.13 Crystal3.1 Shash15.1   SugaR.IC.1    SF 13     #found
-------------------------------------------------------------------------------------------
 1) Nf1-e3           167         37                     8         64        452         5
 2) e5-e6            288                                                                1
 3) Ne4-d6            21                                7          0          5         4
 4) Ra3xc3           535                                                                1
 5) g2-g4              0          0          0          0          0          0         6
 6) .. Qf7x           89                              846                               2
 7) Be4-b1            11                    11                     0          0         4
 8) Rg1-g3            69        229        363          9        534        123         6
 9) Rg4-g6           210        410        223        588                   557         5
10) e2-e4            172         41          2                   534          4         5
11) Re6-e8             5        228         22         42         52         24         6
12) .. Rf8-           84                    80        723          6        872         5
13) Be5xf4             8         30                   391        183        515         5
14) g2-g4             57         16         24         12          6          2         6
15) f5-f6            311                     0          0          0          2         5
16) Ba6-c4            54        161        234        765         36        110         6
17) Nc5-e4                                                       173                    1
18) Rg4-g7           847         12        620                   136                    4
-------------------------------------------------------------------------------------------
#found                17         10         11         12         14         13

The 18 positions :

Code: Select all

5N1n/3P1p1p/5P1k/1Pp1P2p/4KP1P/2p5/p4b2/5N2 w - - bm Ne3; id "The-Engine-Crackers.001" ;
k7/P4pp1/1P6/2p1P3/2P1n3/p5pB/P4n2/K5b1 w - - bm e6; id "The-Engine-Crackers.002" ;
2k5/p2p4/1P6/P3p3/1q2N3/2p5/K7/7Q w - - bm Nd6+; id "The-Engine-Crackers.003" ;
7k/pp1pBp1N/1p3Pp1/b3K1PR/P3P2p/R1n4P/2P1N1P1/3q4 w - - bm Rxc3; id "The-Engine-Crackers.004" ;
2r2r2/p2qnpbk/1pn3p1/2pNp2p/P1P1P3/3P1P2/3N1BPP/R2Q1R1K w - - bm g4; id "The-Engine-Crackers.005" ;
1rb1r3/5qk1/3b2p1/1PnPp1Pp/4B3/pP3Q2/P2N1B1P/1KR3R1 b - - bm Qxf3; id "The-Engine-Crackers.006" ;
3rq2k/1p1r1pp1/p1p1p2p/P1PnP3/3PB1PP/5R1K/3Q1P2/3R4 w - - bm Bb1; id "The-Engine-Crackers.007" ;
2rqkb1r/1b1n1ppp/p1npp3/1p4P1/3NPP2/P1N1B3/1PP4P/R2QKBR1 w Qk - bm Rg3; id "The-Engine-Crackers.008" ;
4nr1k/pp1r2p1/5p1p/2p1PR2/P1B3RQ/3P4/2P3KP/q7 w - - bm Rg6; id "The-Engine-Crackers.009" ;
rnbqk2r/pp1p1ppp/3P1n2/b1p5/8/2N2N2/PP2PPPP/R1BQKB1R w KQkq  - bm e4; id "The-Engine-Crackers.010" ;
R7/8/p3R3/3p4/5n2/K7/4P3/4kn2 w - - bm Ree8; id "The-Engine-Crackers.011" ;
r4r2/b2q2pk/3p2np/N2pp3/PP1P4/2P3PP/1B3P2/R3RQK1 b - - bm Rf5; id "The-Engine-Crackers.012" ;
r7/pp2Pp1k/5P1p/PBp1B1p1/5n2/8/8/5K2 w - - bm Bxf4; id "The-Engine-Crackers.013" ;
rnbq1rk1/4ppb1/p1pp1np1/1p5p/2PPP2P/2N1BP2/PP1Q2P1/2KR1BNR w - - bm g4; id "The-Engine-Crackers.014" ;
K1k5/p2p2p1/2pP4/5PpP/4Pb2/QP4q1/8/8 w - - bm f6; id "The-Engine-Crackers.015" ;
6k1/2p3p1/B4R1P/R3P2K/4p1pP/P3q3/1Qp5/2r4r w - - bm Bc4+; id "The-Engine-Crackers.016" ;
2rqr1k1/pp1bbp1p/2n1p1pB/2N1P3/1PQp4/P4N2/5PP1/3RR1K1 w  - - bm Ne4; id "The-Engine-Crackers.017" ;
3q3N/1p5k/5B2/8/1p2p1Rp/1P5K/p3PP2/8 w - - bm Rg7; id "The-Engine-Crackers.018" ;

7 hardest positions, from hardest to easiest, is something like (it depends on how you rate not found positions) :
4, 2, 17, 6, 18, 9, 12

Code: Select all

7k/pp1pBp1N/1p3Pp1/b3K1PR/P3P2p/R1n4P/2P1N1P1/3q4 w - - bm Rxc3; id "The-Engine-Crackers.004" ;
k7/P4pp1/1P6/2p1P3/2P1n3/p5pB/P4n2/K5b1 w - - bm e6; id "The-Engine-Crackers.002" ;
2rqr1k1/pp1bbp1p/2n1p1pB/2N1P3/1PQp4/P4N2/5PP1/3RR1K1 w  - - bm Ne4; id "The-Engine-Crackers.017" ;
1rb1r3/5qk1/3b2p1/1PnPp1Pp/4B3/pP3Q2/P2N1B1P/1KR3R1 b - - bm Qxf3; id "The-Engine-Crackers.006" ;
3q3N/1p5k/5B2/8/1p2p1Rp/1P5K/p3PP2/8 w - - bm Rg7; id "The-Engine-Crackers.018" ;
4nr1k/pp1r2p1/5p1p/2p1PR2/P1B3RQ/3P4/2P3KP/q7 w - - bm Rg6; id "The-Engine-Crackers.009" ;
r4r2/b2q2pk/3p2np/N2pp3/PP1P4/2P3PP/1B3P2/R3RQK1 b - - bm Rf5; id "The-Engine-Crackers.012" ;

Glarean · Post by **Glarean** » Thu Mar 25, 2021 9:46 am

Thanks for your tests.

They document that the strongest engines show quite different solution behavior, although they have the same creator...

Numbers 3, 5, 7, 10, 15 are now too easy for the latest generation of engines. I guess I will have to remove them from the crackers.

Glarean

Jouni · Post by **Jouni** » Thu Mar 25, 2021 10:27 pm

Vincent one question: Do You run 6 engines at once? If yes which GUI you use?

MikeB · Post by **MikeB** » Fri Mar 26, 2021 2:56 am

Vinvin wrote: ↑Thu Mar 25, 2021 2:25 am In June 2020, Glarean posted 18 "Engine Crackers" positions.
http://talkchess.com/forum3/viewtopic.p ... 25#p846325
All the 8 positions with variations here : https://glarean-magazin.ch/2020/06/12/c ... s-puzzles/

I let run the 5 solvers + SF 13 for 15 minutes (900 seconds) per position. CPU : old i7-6*4GHz, 1 thread used.

On the 18 problems :
Crystal 3.1 solved 11
Bluefish-v13 solved 17 <- best score
Black-Diamond-v13 solved 10
SugaR.AI.ICCF.1.00 solved 14
ShashChess15.1 solved 12
Stockfish-13 solved 13

Table (results are in second unit).

Code: Select all

18 Crackers    Bluefish-13 Black-D.13 Crystal3.1 Shash15.1   SugaR.IC.1    SF 13     #found
-------------------------------------------------------------------------------------------
 1) Nf1-e3           167         37                     8         64        452         5
 2) e5-e6            288                                                                1
 3) Ne4-d6            21                                7          0          5         4
 4) Ra3xc3           535                                                                1
 5) g2-g4              0          0          0          0          0          0         6
 6) .. Qf7x           89                              846                               2
 7) Be4-b1            11                    11                     0          0         4
 8) Rg1-g3            69        229        363          9        534        123         6
 9) Rg4-g6           210        410        223        588                   557         5
10) e2-e4            172         41          2                   534          4         5
11) Re6-e8             5        228         22         42         52         24         6
12) .. Rf8-           84                    80        723          6        872         5
13) Be5xf4             8         30                   391        183        515         5
14) g2-g4             57         16         24         12          6          2         6
15) f5-f6            311                     0          0          0          2         5
16) Ba6-c4            54        161        234        765         36        110         6
17) Nc5-e4                                                       173                    1
18) Rg4-g7           847         12        620                   136                    4
-------------------------------------------------------------------------------------------
#found                17         10         11         12         14         13

The 18 positions :

Code: Select all

5N1n/3P1p1p/5P1k/1Pp1P2p/4KP1P/2p5/p4b2/5N2 w - - bm Ne3; id "The-Engine-Crackers.001" ;
k7/P4pp1/1P6/2p1P3/2P1n3/p5pB/P4n2/K5b1 w - - bm e6; id "The-Engine-Crackers.002" ;
2k5/p2p4/1P6/P3p3/1q2N3/2p5/K7/7Q w - - bm Nd6+; id "The-Engine-Crackers.003" ;
7k/pp1pBp1N/1p3Pp1/b3K1PR/P3P2p/R1n4P/2P1N1P1/3q4 w - - bm Rxc3; id "The-Engine-Crackers.004" ;
2r2r2/p2qnpbk/1pn3p1/2pNp2p/P1P1P3/3P1P2/3N1BPP/R2Q1R1K w - - bm g4; id "The-Engine-Crackers.005" ;
1rb1r3/5qk1/3b2p1/1PnPp1Pp/4B3/pP3Q2/P2N1B1P/1KR3R1 b - - bm Qxf3; id "The-Engine-Crackers.006" ;
3rq2k/1p1r1pp1/p1p1p2p/P1PnP3/3PB1PP/5R1K/3Q1P2/3R4 w - - bm Bb1; id "The-Engine-Crackers.007" ;
2rqkb1r/1b1n1ppp/p1npp3/1p4P1/3NPP2/P1N1B3/1PP4P/R2QKBR1 w Qk - bm Rg3; id "The-Engine-Crackers.008" ;
4nr1k/pp1r2p1/5p1p/2p1PR2/P1B3RQ/3P4/2P3KP/q7 w - - bm Rg6; id "The-Engine-Crackers.009" ;
rnbqk2r/pp1p1ppp/3P1n2/b1p5/8/2N2N2/PP2PPPP/R1BQKB1R w KQkq  - bm e4; id "The-Engine-Crackers.010" ;
R7/8/p3R3/3p4/5n2/K7/4P3/4kn2 w - - bm Ree8; id "The-Engine-Crackers.011" ;
r4r2/b2q2pk/3p2np/N2pp3/PP1P4/2P3PP/1B3P2/R3RQK1 b - - bm Rf5; id "The-Engine-Crackers.012" ;
r7/pp2Pp1k/5P1p/PBp1B1p1/5n2/8/8/5K2 w - - bm Bxf4; id "The-Engine-Crackers.013" ;
rnbq1rk1/4ppb1/p1pp1np1/1p5p/2PPP2P/2N1BP2/PP1Q2P1/2KR1BNR w - - bm g4; id "The-Engine-Crackers.014" ;
K1k5/p2p2p1/2pP4/5PpP/4Pb2/QP4q1/8/8 w - - bm f6; id "The-Engine-Crackers.015" ;
6k1/2p3p1/B4R1P/R3P2K/4p1pP/P3q3/1Qp5/2r4r w - - bm Bc4+; id "The-Engine-Crackers.016" ;
2rqr1k1/pp1bbp1p/2n1p1pB/2N1P3/1PQp4/P4N2/5PP1/3RR1K1 w  - - bm Ne4; id "The-Engine-Crackers.017" ;
3q3N/1p5k/5B2/8/1p2p1Rp/1P5K/p3PP2/8 w - - bm Rg7; id "The-Engine-Crackers.018" ;

7 hardest positions, from hardest to easiest, is something like (it depends on how you rate not found positions) :
4, 2, 17, 6, 18, 9, 12

Code: Select all

7k/pp1pBp1N/1p3Pp1/b3K1PR/P3P2p/R1n4P/2P1N1P1/3q4 w - - bm Rxc3; id "The-Engine-Crackers.004" ;
k7/P4pp1/1P6/2p1P3/2P1n3/p5pB/P4n2/K5b1 w - - bm e6; id "The-Engine-Crackers.002" ;
2rqr1k1/pp1bbp1p/2n1p1pB/2N1P3/1PQp4/P4N2/5PP1/3RR1K1 w  - - bm Ne4; id "The-Engine-Crackers.017" ;
1rb1r3/5qk1/3b2p1/1PnPp1Pp/4B3/pP3Q2/P2N1B1P/1KR3R1 b - - bm Qxf3; id "The-Engine-Crackers.006" ;
3q3N/1p5k/5B2/8/1p2p1Rp/1P5K/p3PP2/8 w - - bm Rg7; id "The-Engine-Crackers.018" ;
4nr1k/pp1r2p1/5p1p/2p1PR2/P1B3RQ/3P4/2P3KP/q7 w - - bm Rg6; id "The-Engine-Crackers.009" ;
r4r2/b2q2pk/3p2np/N2pp3/PP1P4/2P3PP/1B3P2/R3RQK1 b - - bm Rf5; id "The-Engine-Crackers.012" ;

Try Bluefish 13 on id "The-Engine-Crackers.002" with this net
https://tests.stockfishchess.org/api/nn ... 3107c.nnue

one thread , 256 Hash
dep score nodes time (not shown: tbhits knps seldep)

Code: Select all

 34	+3.99!	812.1M	3:58.88	e6!
 33	3 minutes
 33	2 minutes
 33	110 seconds
 33	100 seconds
 33	90 seconds
 33	80 seconds
 33	70 seconds
 33	60 seconds
 33	50 seconds
 33	40 seconds
 33	30 seconds
 33	+3.87 	504.0M	2:30.98	e6 fxe6 Bg2 Kb7 Kb1 Kxb6 a8=Q Kc7 Kc2 Nf6 Qc6+ Kd8 Qxc5 Ke8 Qa7 g5 Bc6+ Kd8 Qb8+ Ke7 Qc7+ Kf8 Qxg3 Nh3 Qxa3+ Kf7 Qxh3 Bd4 a4 e5 Qd3 Ke7 a5 Kd6 Ba8 Ke6 a6 Ng4 Bd5+ Kd7 Qf5+ Kd6 Qxg4
 33	+3.81!	367.2M	1:53.70	e6!
 33	+3.56!	210.7M	1:08.11	e6!
 33	+3.36!	141.9M	0:47.49	e6!
 33	+3.22!	104.7M	0:35.66	e6!
 33	+3.12!	79.3M  	0:27.31	e6!
 32	20 seconds
 32	+3.02 	62.8M  	0:22.02	e6 fxe6 Bg2 Kb7 Kb1 Kxb6 a8=Q Kc7 Kc2 Kd6 Bxe4 Nxe4 Qxe4 Bd4 Kb3 Bb2 Qf3 Ke5 Qxg3+ Ke4 Qg4+ Kd3 Qg6+ Kd2 Qg5+ Kd3 Qxc5 Bd4 Qd6 e5
 32	+2.48!	55.0M  	0:19.63	e6!
 32	+1.86!	52.6M  	0:18.90	e6!
 32	+1.37!	51.6M  	0:18.61	e6!
 32	+1.00!	50.7M  	0:18.36	e6!
 32	+0.71!	49.9M  	0:18.13	e6!
 32	+0.48!	48.4M  	0:17.66	e6!
 32	+0.32!	46.9M  	0:17.20	e6!
 32	+0.20!	45.2M  	0:16.65	e6!
 32	+0.12!	43.8M  	0:16.20	e6!
 32	+0.06!	41.8M  	0:15.52	e6!
 31	10 seconds
 31	  0.00 	18.6M  	0:06.98	Bd7 Kb7 Bc6+ Kxc6 a8=Q+ Kxb6 Qb8+ Ka6 Qb5+ Ka7 Qd7+ Kb6 Qd8+
 30	  0.00 	15.8M  	0:05.93	Bd7 Kb7 Bc6+ Kxc6 a8=Q+ Kxb6 Qb8+ Ka6 Qa8+
 29	  0.00 	12.2M  	0:04.58	Bd7 Kb7 Bc6+ Kxc6 a8=Q+ Kxb6 Qb8+ Ka6 Qa8+
 28	  0.00 	10.2M  	0:03.86	Bd7 Kb7 Bc6+ Kxc6 a8=Q+ Kxb6 Qb8+ Ka6 Qa8+
 27	  0.00 	8.01M  	0:03.05	Bd7 Kb7 Bc8+ Ka8
 26	  0.00 	6.72M  	0:02.59	Bd7 Kb7 Bc8+ Ka8 Bd7
 25	  0.00 	3.08M  	0:01.19	Bd7 Kb7 Bc6+ Kxc6 a8=Q+ Kxb6 Qb8+ Kc6 Qc8+ Kb6
 24	  0.00 	2.02M  	0:00.78	Bd7 Kb7 Bc8+ Ka8
 23	0 seconds
 23	  0.00 	1.28M  	0:00.47	Bd7 Kb7 Bc8+ Ka8
 22	  0.00 	1.06M  	0:00.40	Bd7 Kb7 Bc8+ Ka8
 21	  0.00 	900412	0:00.34	Bd7 Kb7 Bc8+ Ka8
 20	  0.00 	748228	0:00.28	Bd7 Kb7 Bc8+ Ka8
 19	  0.00 	350650	0:00.13	Bd7 Kb7 Bc8+ Ka8
 18	  0.00 	240635	0:00.09	Bd7 Kb7 Bc8+ Ka8
 17	  0.00 	184621	0:00.07	Bd7 Kb7 Bc8+ Ka8 Bd7
 16	  0.00 	98834  	0:00.04	Bd7 Kb7 Bc6+ Kxc6 a8=Q+ Kxb6 Qb8+ Kc6 Qc8+ Kb6
 15	  0.00 	60726  	0:00.03	Bd7 Kb7 Bc8+ Ka8
 14	  0.00 	37704  	0:00.02	Bd7 Kb7 Bc6+ Kxc6 a8=Q+ Kxb6 Qd8+ Ka6 Qa8+
 13	  0.00 	30176  	0:00.01	Bd7 Kb7 Bc6+ Kxc6 a8=Q+ Kxb6 Qd8+ Ka6 Qa8+
 12	  0.00 	25660  	0:00.01	Bd7 Kb7 Bc6+ Kxc6 a8=Q+ Kxb6 Qd8+ Ka6 Qa8+
 11	  0.00 	8222    	0:00.01	Bd7 Kb7 Bc6+ Kxc6 a8=Q+ Kxb6 Qb8+ Ka6 Qa8+
 10	  0.00 	4929    	0:00.00	Bd7 Kb7 Bc6+ Kxc6 a8=Q+ Kxb6 Qb8+ Ka6 Qc8+ Ka7 Qc7+ Ka6
  9	  0.00 	3013    	0:00.00	Bd7 Kb7 Bc6+ Kxc6 a8=Q+ Kxb6 Qb8+ Kc6 Qc8+ Kb6
  8	 -0.02 	2023    	0:00.00	Bd7 Kb7 Bc6+ Kxc6 a8=Q+ Kxb6 Qb8+ Kc6
  7	 -0.28 	1675    	0:00.00	Bd7 Kb7 Bc6+ Kxc6 a8=Q+ Kxb6 Qxa3
  6	 -0.24 	1388    	0:00.00	Bd7 Kb7
  5	 -2.57 	1203    	0:00.00	Bd7 Kb7
  4	 -3.36 	292      	0:00.00	Bg2 Kb7 Bf3 Bh2
  3	 -3.42 	76        	0:00.00	Bg2 g6 Bf1
  2	 -2.87 	41        	0:00.00	Bg2 g6
  1	NNUE evaluation using C:\cluster.mfb\Popcnt-LP\eval\nn-1f660bd3107c.nnue enabled
  1	Found 510 tablebases
  1	 -3.12 	22        	0:00.00	Bg2

Marek Soszynski · Post by **Marek Soszynski** » Fri Mar 26, 2021 1:05 pm

Vinvin wrote: ↑Thu Mar 25, 2021 2:25 am I let run the 5 solvers + SF 13 for 15 minutes (900 seconds) per position. CPU : old i7-6*4GHz, 1 thread used.

On the 18 problems :
Crystal 3.1 solved 11
Bluefish-v13 solved 17 <- best score
Black-Diamond-v13 solved 10
SugaR.AI.ICCF.1.00 solved 14
ShashChess15.1 solved 12
Stockfish-13 solved 13

SugaR and ShashChess have recently been updated. How would they do now? And BlueFish with the alternative net?

Vinvin · Post by **Vinvin** » Fri Mar 26, 2021 1:39 pm

Jouni wrote: ↑Thu Mar 25, 2021 10:27 pm Vincent one question: Do You run 6 engines at once? If yes which GUI you use?

I run 3 engines at once with Arena :
Start the interface 3 times. In the "output" field, set different names for the output files.

A look back at the Engine Crackers

A look back at the Engine Crackers

Re: A look back at the Engine Crackers

Re: A look back at the Engine Crackers

Re: A look back at the Engine Crackers

Re: A look back at the Engine Crackers

Re: A look back at the Engine Crackers