ERET-Test-Suite: New Results

[Glarean]

The ERET Chess Test is a collection of 111 easy and difficult puzzles for bad and good chess programs.
Its first release was in spring 2017.

The ERET collection has the job to test the playing strength of new chess programs within half an hour at the most.

Therefore it contains 111 sample positions which cover a very large range of chess positions.
The rankings determined with the ERET therefore correspond quite exactly with the rankings of the usual computer chess tournaments (e.g. CCRL and many others).

I have now (again) tested two dozen engines with all 111 ERET positions (15sec/position). (The selection of the engines was more or less random and had to contain strong and medium strong and weak programs).

My current ERET Ranking looks like this:

Code: Select all

Engine              Solutions

Stockfish 9         77/111
Houdini 6.03        76/111
Komodo 12.1.1       75/111
Ethereal 11.12      66/111
Deep Shredder 13    61/111
Booot 6.3           57/111
Andscacs 0.94       54/111
LC0 18.1 (Cuda)     54/111
Fritz 16            53/111
Equinox 3.3         50/111
Critter 1.6a        49/111
Gull 3.1            45/111
Chiron 4.4          41/111
Naum 4.6            40/111
Wasp 3.0            37/111
BlackMamba 2.0      34/111
Deep Fritz 10       23/111
Crafty 25.2         22/111
SOS 5.1             21/111
Minko 1.3           18/111
Onno 1.2.7          17/111
Alfil 13.1          17/111
Monarch 1.7         14/111
Clueless 1.4        11/111

Here you can download all files (PGN / EPD / CBH / XLS) and see the detailed setting of the whole test:
https://glarean-magazin.ch/2017/03/05/c ... test-eret/
(For a translation click the button "Translate" below left).

Have fun with your own tests!

Greetings: Walter

.

[Jouni]

SF 141018 with 4 cores i5 and 512 MB hash 15 seconds: 1. try 82 and second 80.

[Glarean]

My current ERET Ranking looks like this:

Code: Select all

Engine              Solutions

Stockfish 9         77/111
Houdini 6.03        76/111
Komodo 12.1.1       75/111
Ethereal 11.12      66/111
Deep Shredder 13    61/111
Booot 6.3           57/111
Andscacs 0.94       54/111
LC0 18.1 (Cuda)     54/111
Fritz 16            53/111
Equinox 3.3         50/111
Critter 1.6a        49/111
Gull 3.1            45/111
Chiron 4.4          41/111
Naum 4.6            40/111
Wasp 3.0            37/111
BlackMamba 2.0      34/111
Deep Fritz 10       23/111
Crafty 25.2         22/111
SOS 5.1             21/111
Minko 1.3           18/111
Onno 1.2.7          17/111
Alfil 13.1          17/111
Monarch 1.7         14/111
Clueless 1.4        11/111

Here you can download all files (PGN / EPD / CBH / XLS) and see the detailed setting of the whole test:
https://glarean-magazin.ch/2017/03/05/c ... test-eret/
(For a translation click the button "Translate" below left).

The (correct) newer link for all ERET downloads (sorry):
https://glarean-magazin.ch/2018/10/17/e ... tellungen/

.

[Vinvin]

Hi, Walter !
Is there the same positions as here : http://rybkaforum.net/cgi-bin/rybkaforu ... #pid573015 ?

Vincent

[Glarean]

Hi, Walter !
Is there the same positions as here : http://rybkaforum.net/cgi-bin/rybkaforu ... #pid573015 ?
Vincent

Hi Vincent
Yes. (However, my tests ran without 6-syzygy tablebases and with 15 seconds per position).
Walter

[Dann Corbit]

There is one difficult problem in the set:
[d]2b1r3/r2ppN2/8/1p1p1k2/pP1P4/2P3R1/PP3PP1/2K5 w - -
I think that all three of these plans are draws:
2b1r3/r2ppN2/8/1p1p1k2/pP1P4/2P3R1/PP3PP1/2K5 w - - pv Rf3+ Ke6 Nh6 Rh8 Ng4 d6 a3 Ra8 Kd2 Bb7 Re3+ Kd7 Ke2 Rhg8 f3 Raf8 Rd3 Kc6 Ne3 Bc8 Kf2 Rg7 Rd1 Rh8 Kg1 Rgg8 Kf2 Rh4 Rd2 Bd7 Rd1 Ra8 Kg3 Rah8 Kf2 Rh1 Rd2 Ra8 Ke2 Rf8 Kf2 Rhh8 Rd1 e6 Ke2 Rh7 Kf2 Re8 Kg3 Reh8 Kf2 Rh1 Rd2 Rg8;

2b1r3/r2ppN2/8/1p1p1k2/pP1P4/2P3R1/PP3PP1/2K5 w - - pv Nh6+ Kf6 Ng4+ Ke6 Re3+ Kf5 Rg3 d6 Ne3+ Ke6 Rh3 Rf8 f3 Bb7 Rh1 Raa8 Re1 Rh8 Kc2 Kf7 a3 Rag8 Rf1 Rh4 Kd1 e6 Ke2 Bc6 Ng4 Be8 Rg1 Bd7 Rd1 Rh5 Ne3 Rgh8 Kf2 Rh4 Rd2 Bc6 Ng4 Rh1 Ke3 Rg8 Re2 Bd7 Kd2 Rh7 Ne3 Rh4 Kd3 Rh1 Kd2 Bc6 Ng4 Rh4 Kd3 Ke7 Rd2 Be8 Ne3 Rh1 Ke2 Rgh8 Kf2 Bg6;

2b1r3/r2ppN2/8/1p1p1k2/pP1P4/2P3R1/PP3PP1/2K5 w - - pv Nd6+ exd6 Rf3+ Ke6 Re3+ Kf7 Rf3+ Ke7 Re3+ Kd8 Rxe8+ Kxe8 a3 Rc7 Kd2 Bb7 Ke3 Rc4 f3 Rc8;

Although I agree that Nd6+ is the most obviously correct and the only one that is a lock to be a draw (the others had repeated scores for many iterations but took a very long time and the cycles are quite deep).

Sting 10 is the only engine I tried that nailed this problem quickly.

Code: Select all

 1 
 Avoid move: 
 Best move (Sting10): Nf7-d6
 Not found in: 57:00
 16	00:00	 1,952,190	2,833,367	-3.75	Re3 Ra6 Rf3+ Ke6 g4 Rg8 Nh6 Rg6 Re3+ Kf6 Rf3+ Kg5 Nf7+ Kh4 g5 a3 bxa3 d6 Rf4+ Bg4
 17-	00:00	 2,691,811	3,424,695	-3.83	Re3 Ra6
 17-	00:00	 3,130,387	3,717,799	-3.91	Re3 Ra6
 17	00:00	 3,746,075	4,054,193	-3.83	Nh6+ Kf6 Ng4+ Ke6 Ne3 d6 Rg6+ Kf7 Rh6 Kg7 Rh4 e6 f3 Rg8 Kd2 Kf6 g4
 18-	00:01	 4,593,448	4,485,789	-3.95	Nh6+ Kf6
 18	00:01	 6,123,821	5,146,068	-4.04	Nh6+ Kf6 Ng4+ Ke6 Ne3 d6 Kd2 Rh8 f4 Bd7 f5+ Kf7 Rg5 Bc6 Rg6 Raa8 Rg5 a3 b3
 19	00:01	 9,545,895	6,239,147	-4.08	Nh6+ Kf6 Ng4+ Ke6 Ne3 d6 Rg6+ Kf7 Rh6 Kg7 Rh4 Be6 f4 Rf8 Kc2 Raa8 Kd3 a3 b3 Kg6 g3 Rac8 Rh1
 20+	00:01	 14,566,417	7,334,550	-3.95	Rf3+
 20+	00:02	 14,973,577	7,383,420	-3.83	Rf3+ Ke6 Nh6 a3 g4 axb2+ Kb1 d6 g5 Rh8 Re3+ Kd7 g6 Rxh6 g7 Rg6 Rg3 Rxg3 fxg3 Ke6 g8Q+ Kf6 Qxc8 Kg5 Kxb2 Kf6 g4 Kg5 Kb1 Kf6
 20	00:02	 16,012,435	7,382,404	-3.83	Rf3+ Ke6 g4 a3 Nh6 axb2+
 21-	00:02	 19,891,704	7,749,008	-3.95	Rf3+ Ke6
 21-	00:02	 22,914,630	7,839,421	-4.08	Rf3+ Ke6 g4 Rg8
 21	00:03	 26,305,926	8,081,697	-4.00	Re3 Ra6 Rf3+ Ke6 g4 Rg8 Nh6 Rg6 Re3+ Kf6 Rg3 Rg7 Nf5 Rh7 Rf3 Kg6 Ne3 Bb7 Kd2 a3 b3 e6
 22-	00:03	 32,342,345	8,471,017	-4.12	Re3 Ra6
 22	00:04	 42,775,419	8,950,704	-4.08	Nh6+ Kf6 Ng4+ Ke6 Ne5 d6 Nd3 Kd7 Rg7 Bb7 Nf4 Rf8 g3 Raa8 Kd2 a3 b3 Rg8 Rf7 Raf8 Rxf8 Rxf8
 23-	00:05	 47,389,533	9,052,441	-4.20	Nh6+ Kf6
 23+	00:06	 56,988,666	9,336,282	-3.95	Nh6+ Kf6 Ng4+ Ke6 Ne3 d6 Rg6+ Kf7 Rh6 Kg7 Rh1 Be6 Kd2 Rf8 f3 Raa8 Rf1 Rh8 f4 a3 b3 Kf7 f5 Rh1 Rxh1 Rh8 Rxh8
 23	00:06	 60,184,066	9,279,072	-4.20	Nh6+ Kf6 Ng4+ Ke6 Ne3 d6 Rg6+ Kf7 Rh6 Kg7 Rh1 Rf8 f3 Be6 Kd2
 24	00:07	 69,509,668	9,528,398	-4.20	Nh6+ Kf6 Ng4+ Ke6 Ne3 d6 Rg6+ Kf7 Rh6 Kg7 Rh1
 25	00:09	 94,618,131	9,855,028	-4.24	Nh6+ Kf6 Ng4+ Ke6 Ne3 d6 Rg6+ Kf7 Rh6 Kg7 Rh1 Rf8 f3 Be6 Rf1 a3 b3 Rc7 Kd2 Rfc8 Rc1 Kf6 f4 Bf7 Ng4+ Kg6 Ne3 e6 g4 Kf6 g5+ Kg6 Ng4 Rh8 Re1
 26+	00:13	 132,837,021	10,117,061	-4.16	Re3
 26	00:15	 156,156,872	10,231,073	-4.28	Re3 Ra6 Rf3+ Ke6 Ne5 Rg8 Nd3 Kd6 g3 a3 b3 Ra8 Ne5 Bb7 Kd2 Rg7 Rf5 Rh8 g4 Ke6 Rh5 Rxh5 gxh5 Rg2 Ke3 Rh2 Ng6 Rh3+ Kd2
 27+	00:21	 234,085,203	10,669,334	-4.20	Rf3+
 27+	00:22	 237,154,902	10,656,252	-4.12	Rf3+
 27	00:25	 268,171,450	10,705,874	-4.28	Nh6+
 28-	00:35	 383,054,985	10,851,108	-4.36	Nh6+ Kf6
 28	00:42	 465,117,177	10,845,431	-4.36	Nh6+ Kf6 Ng4+ Ke6 Ne3 d6 Rh3 Rf8 f3 Bb7 Rh1 Raa8 Rf1 Kf6 f4 Rh8 Kd2
 29+	00:56	 625,817,743	11,059,977	-4.28	Re3
 29	01:05	 728,722,904	11,108,411	-4.36	Nh6+
 30	01:22	 917,495,924	11,149,949	-4.40	Re3 Ra6 Rf3+ Ke6 Ne5 Rg8 Nd3 Kd6 Nf4 a3 b3 Rg4 Kd2 Bb7 g3 Ra8 Nd3 Rg7 Re3 Rh8 Ne5 Rh1 Re2 Ra1 Kd3 Rd1+ Kc2 Rf1 Kd3 Rg8 g4
 31+	01:25	 949,662,508	11,129,683	-4.32	Re3
 31	01:42	1,148,565,019	11,182,819	-4.32	Nd6+ exd6 Rf3+ Ke6 Re3+ Kf7 Rf3+ Ke7 Re3+ Kd8 Rxe8+ Kxe8 a3 Rc7 Kd2 Ke7 f3 Ba6 Ke3 Rc8 Kf2 Kf6 g3 Rg8 Kg2 Re8 Kf1 Kg6 Kf2 Kf5 Kf1 Bb7 Kf2 Rh8 Kg2 Kg5 Kg1 Kf6 Kg2
 32	02:00	1,350,406,345	11,211,809	-4.32	Nd6+ exd6 Rf3+ Ke6 Re3+ Kf7 Rf3+ Ke7 Re3+ Kd8 Rxe8+ Kxe8 a3 Rc7 Kd2 Ke7 f3 Ba6 Ke3 Rc8 Kf2
 33	02:15	1,528,156,630	11,246,782	-4.32	Nd6+ exd6 Rf3+ Ke6 Re3+ Kf7 Rf3+ Ke7 Re3+ Kd8 Rxe8+ Kxe8 a3 Rc7 Kd2 Ke7 f3 Ba6 Ke3 Rc8 Kf2 Kf6 g3 Rg8 Kg2 Re8 Kf1
 34	02:46	1,881,495,708	11,326,805	-4.32	Nd6+ exd6 Rf3+ Ke6 Re3+ Kf7 Rf3+ Ke7 Re3+ Kd8 Rxe8+ Kxe8 a3 Rc7 Kd2 Ke7 f3 Ba6 Ke3 Kf6 g3 Ke7 Kf2 Rc8 Kg2 Rh8 Kg1 Rb8 Kf1 Re8 Kg1 Kf6 Kf2 Rg8 Kg2 Bc8 Kf2 Bb7 Kg2 Rb8
 35	03:17	2,253,668,667	11,390,276	-4.32	Nd6+ exd6 Rf3+ Ke6 Re3+ Kf7 Rf3+ Ke7 Re3+ Kd8 Rxe8+ Kxe8 a3 Rc7 Kd2 Ke7 f3 Ba6 Ke3 Kf6 g3 Ke7 Kf2 Rc8 Kg2 Rh8 Kg1 Ke6 Kg2 Rc8
 36	04:23	3,032,664,140	11,489,409	-4.32	Nd6+ exd6 Rf3+ Ke6 Re3+ Kf7 Rf3+ Ke7 Re3+ Kd8 Rxe8+ Kxe8 a3 Rc7 Kd2 Ke7 f3 Ba6 Ke3 Kf6 g3 Ke7 Kf2 Rc8 Kg2 Rh8 Kg1 Kf6 Kg2 Rg8 Kf2 Bb7 Kg2 Rg7 Kf2 Re7 Kf1 Kf5 Kf2 Ba6 Kf1 Re8 Kf2 Rg8
 37	05:34	3,859,993,009	11,527,081	-4.32	Nd6+ exd6 Rf3+ Ke6 Re3+ Kf7 Rf3+ Ke7 Re3+ Kd8 Rxe8+ Kxe8 a3 Rc7 Kd2 Ke7 f3 Ba6 Ke3 Kf6 g3 Ke7 Kf2 Rc8 Kg2 Rh8 Kg1 Kf6 Kg2 Rg8 Kf2 Kg5 Kf1 Rh8 Kg2 Bb7 Kg1 Bc8 Kg2 Ba6 Kg1 Kh6
 38	06:56	4,822,181,825	11,583,095	-4.32	Nd6+ exd6 Rf3+ Ke6 Re3+ Kf7 Rf3+ Ke7 Re3+ Kd8 Rxe8+ Kxe8 a3 Rc7 Kd2 Bb7 Ke3 Rc4 f3 Rc8 Kf2 Ke7 Kf1 Rf8 Kf2 Rg8 g3 Rc8 Kg2 Kf6 Kf2 Kf5 Kf1 Rf8 Kg1 Rh8
 39	11:17	7,890,340,843	11,653,087	-4.32	Nd6+ exd6 Rf3+ Ke6 Re3+ Kf7 Rf3+ Ke7 Re3+ Kd8 Rxe8+ Kxe8 a3 Rc7 Kd2 Bb7 Ke3 Rc4 f3 Rc8 Kf2 Ke7 Kf1 Rf8 Kf2 Rg8 g3 Rh8 Kg2 Kf6 Kg1 Ke7
 40	15:04	10,583,454,890	11,700,411	-4.32	Nd6+ exd6 Rf3+ Ke6 Re3+ Kf7 Rf3+ Ke7 Re3+ Kd8 Rxe8+ Kxe8 a3 Rc7 Kd2 Bb7 Ke3 Rc4 f3 Rc8 Kf2 Ke7 Kf1 Rf8 Kf2 Rg8 g3 Rh8 Kg2 Kf7 Kg1 Re8 Kf2 Kf6 Kf1 Kf5 Kf2 Rg8 Kg2 Rh8 Kg1 Kg5 Kg2 Kg6 Kg1 Kf5
 41	20:51	14,712,164,807	11,752,451	-4.32	Nd6+ exd6 Rf3+ Ke6 Re3+ Kf7 Rf3+ Ke7 Re3+ Kd8 Rxe8+ Kxe8 a3 Rc7 Kd2 Bb7 Ke3 Rc4 f3 Rc8
 42	26:10	18,544,883,686	11,810,365	-4.32	Nd6+ exd6 Rf3+ Ke6 Re3+ Kf7 Rf3+ Ke7 Re3+ Kd8 Rxe8+ Kxe8 a3 Rc7 Kd2 Bb7 Ke3 Rc4 f3 Rc8 Kf2 Ra8 g3 Ke7 Kg1 Kf6 Kf1 Rc8 Kf2 Rh8 Kg1 Kf5 Kg2 Rg8 Kf2 Ra8 Kg1 Rc8 Kf2 Re8 Kf1 Kg5 Kf2 Kg6 Kf1 Rh8 Kg1 Kg5 Kg2 Kf5 Kg1
 43	38:55	27,739,237,966	11,875,156	-4.32	Nd6+ exd6 Rf3+ Ke6 Re3+ Kf7 Rf3+ Ke7 Re3+ Kd8 Rxe8+ Kxe8 a3 Rc7 Kd2 Bb7 Ke3 Rc4 f3 Rc8 Kf2 Ra8 g3 Ke7 Kg2 Kf6 Kf2 Rh8 Kg2 Rg8 Kf2 Kf7
 2018-10-19 7:50:25 AM, Time for this analysis: 00:57:00, Rated time: 57:00

[Glarean]

There is one difficult problem in the set:
[d]2b1r3/r2ppN2/8/1p1p1k2/pP1P4/2P3R1/PP3PP1/2K5 w - -
I think that all three of these plans are draws:
2b1r3/r2ppN2/8/1p1p1k2/pP1P4/2P3R1/PP3PP1/2K5 w - - pv Rf3+ Ke6 Nh6 Rh8 Ng4 d6 a3 Ra8 Kd2 Bb7 Re3+ Kd7 Ke2 Rhg8 f3 Raf8 Rd3 Kc6 Ne3 Bc8 Kf2 Rg7 Rd1 Rh8 Kg1 Rgg8 Kf2 Rh4 Rd2 Bd7 Rd1 Ra8 Kg3 Rah8 Kf2 Rh1 Rd2 Ra8 Ke2 Rf8 Kf2 Rhh8 Rd1 e6 Ke2 Rh7 Kf2 Re8 Kg3 Reh8 Kf2 Rh1 Rd2 Rg8;
2b1r3/r2ppN2/8/1p1p1k2/pP1P4/2P3R1/PP3PP1/2K5 w - - pv Nh6+ Kf6 Ng4+ Ke6 Re3+ Kf5 Rg3 d6 Ne3+ Ke6 Rh3 Rf8 f3 Bb7 Rh1 Raa8 Re1 Rh8 Kc2 Kf7 a3 Rag8 Rf1 Rh4 Kd1 e6 Ke2 Bc6 Ng4 Be8 Rg1 Bd7 Rd1 Rh5 Ne3 Rgh8 Kf2 Rh4 Rd2 Bc6 Ng4 Rh1 Ke3 Rg8 Re2 Bd7 Kd2 Rh7 Ne3 Rh4 Kd3 Rh1 Kd2 Bc6 Ng4 Rh4 Kd3 Ke7 Rd2 Be8 Ne3 Rh1 Ke2 Rgh8 Kf2 Bg6;
2b1r3/r2ppN2/8/1p1p1k2/pP1P4/2P3R1/PP3PP1/2K5 w - - pv Nd6+ exd6 Rf3+ Ke6 Re3+ Kf7 Rf3+ Ke7 Re3+ Kd8 Rxe8+ Kxe8 a3 Rc7 Kd2 Bb7 Ke3 Rc4 f3 Rc8;
Although I agree that Nd6+ is the most obviously correct and the only one that is a lock to be a draw (the others had repeated scores for many iterations but took a very long time and the cycles are quite deep).

Thanks for your analysis. I will take a closer look at your draw suggestions.

Regards: Walter

.