EGTB Testsuite

[Kirill Kryukov]

Hi everyone! I realized that we are still lacking reliable way to test whether an engine is using endgame tablebases or not. We need a way to test if an engine is actually accessing the tablebases. So I propose to create a set of positions for testing each piece configuration (or as many as possible).

Each position should satisfy these conditions:
1. White to move.
2. Position should be a few moves before the endgame it is testing (so some captures will follow to convert into the EGTB position). This is needed to test that an engine is really accessing the EGTB in search, not just in root.
3. Position should have only one winning move, or only one drawing move if there are no winning moves. This requirement is needed to make this test suite useful for checking bitbases too.
4. Position should be trivial to solve for an engine that probes the relevant table in its search. Finding correct move should take a few seconds for an engine accessing the endgame table being tested.
5. Position should be hard to solve without that particular table it is testing. An engine that does not have access to the table should prefer wrong moves, even if it has all other tables (with the same number of pieces).
6. Solution should be correct considering the 50 moves rule. (So that the test will be valid for both current Nalimov tables that ignore 50 moves rule and for future tables that will take it into account).
7. Position should be legal, if possible.

As you can see it is not a trivial task to construct a perfect test position for an endgame table. If you can manage to create any positions, please post here! For a start, may be some positions for testing 4-men or 5-men tables can be made.

Eventually we should have a test position for each piece configuration, so running an engine through the test suite will reveal any missing tables and test whether that engine is able to use them. Note that this test would be useful not only for Nalimov tables, but for any other format as well.

Any ideas or suggestions, and of course any positions are welcome!

Thanks!

(Crossposting from here, because I feel this is of general interest and can be useful for many)

[F. Bluemers]

using some tools you might relax on some conditions:
http://technet.microsoft.com/nl-nl/sysi ... n-us).aspx
Best
Fonzy

[Mike S.]

But basically, this is easy to tell without any specific test positions, because in a late endgame position (i.e. 8 or 7 pieces):

1. using an engine which does access Nalimovs will cause drive activity (HD or USB, most of these devices will tell it by drive LEDs flashing)

2. using an engine which does not access Nalimovs will not cause drive activity.

(I realise that it is more difficult to tell with bitbases)

Anyway, I have a good test position but only if you use 3- and 4-piece Nalimovs only:

[D]7k/4K1n1/8/6P1/8/3B4/8/8 w - - 0 1

This is a study by Sachodjakin. ONLY 1.Bg6 wins. Even with 3- and 4-piece tbs. present, many engines accessing them have trouble to realise that. But with 5-piece tbs., they will of course immediatly choose 1.Bg6. You could run multi-pv with 2 variations to see the difference clearly. - Attention: For that test, you need to switch off the GUI (interface) access for tablebases if available, like in Fritz. Because we want to know what the engines do (only), in this case.

[Kirill Kryukov]

But basically, this is easy to tell without any specific test positions, because in a late endgame position (i.e. 8 or 7 pieces):

1. using an engine which does access Nalimovs will cause drive activity (HD or USB, most of these devices will tell it by drive LEDs flashing)

2. using an engine which does not access Nalimovs will not cause drive activity.

(I realise that it is more difficult to tell with bitbases)

Yes, it can be difficult with in-memory bitbases or when you run engine on a remote computer. Also engine may access other tables, you can't easily see whether it is accessing particular table or not. Also disk cache can hide disk access.

Anyway, I have a good test position but only if you use 3- and 4-piece Nalimovs only:

[D]7k/4K1n1/8/6P1/8/3B4/8/8 w - - 0 1

This is a study by Sachodjakin. ONLY 1.Bg6 wins. Even with 3- and 4-piece tbs. present, many engines accessing them have trouble to realise that. But with 5-piece tbs., they will of course immediatly choose 1.Bg6. You could run multi-pv with 2 variations to see the difference clearly. - Attention: For that test, you need to switch off the GUI (interface) access for tablebases if available, like in Fritz. Because we want to know what the engines do (only), in this case.

Thanks, this is very nice! Not perfect unfortunately:

1. This position itself is contained in the table it is testing. So it does not test whether engine's search probes this table.

2. When I test with all 5-men tables except KBPKN, many engines can still find solution: Rybka 2.3.2a: 4 sec, Chess Tiger 2007.1: 4 sec, Aristarch 4.50: 5 sec, Crafty 21.5: 8 sec.

3. Some engines solve it quickly without using any tablebases: Movei 00.8.438: 1 sec, Glaurung 2.0.1: 2 sec, Chess Tiger 2007.1: 5 sec, Naum 2.0: 6 sec.

BTW I realize that it is very hard to find a good endgame test. May be it is not possible to find any meaningful number of such positions. If so then may be I will try to construct a simpler test-suite with more relaxed requirements.

Best,
Kirill

[Kirill Kryukov]

using some tools you might relax on some conditions:
http://technet.microsoft.com/nl-nl/sysi ... n-us).aspx
Best
Fonzy

Which tool? Which condition? Thanks.

[Werner]

Hii Kirill,
to see whether a new engine is installed correctly and uses tbs or egbbs I normally use this position:
[D]4k3/4p3/8/8/3p4/8/4N3/4KB2 w - -

engines with egbbs responds with high value:
14/24 0:01 +52.51 1.Sxd4 Kd7 2.Ke2 Kd6 3.Kd3 Kd5 4.Ke3 e6 (1.683.792) 1

engines with egtbs in search responds with a mate score:
6.11 0:00 +M29 1.Sxd4 e5 2.Se6 Ke7 3.Lc4 (460) 46
Bester Zug: Se2xd4 Zeit: 0:00.625 min K/s: 46.000 Knoten: 460

and engines with no access with about +6:
Engine: Ktulu 8 (64 MB)
von Rahman Paidar
12.00 0:01 +6.97 1.Sxd4 e5 2.Sf5 e4 3.Lc4 e3 4.Ke2 Kd7 5.Ld5 Kc7 6.Le6 Kb6 7.Lc4 Kc7 8.Ld5 Kd7 9.La2 Kc7 10.Lf7 Kc6 (2.657.411) 1736

[F. Bluemers]

file mon and process monitor.
they will show you if and which (tb) files are accessed,and if its the engine or the gui accessing them.

Best
Fonzy

[smirobth]

Here is a position that is solved instantly with the R+P vs R tablebase, but no programs solve it without them:

[D]k7/P6R/3K4/8/7P/2r5/8/8 w - - 0 1
Rh8+! is the only way to win.

[George Tsavdaris]

Here is a position that is solved instantly with the R+P vs R tablebase, but no programs solve it without them:

[D]k7/P6R/3K4/8/7P/2r5/8/8 w - - 0 1
Rh8+! is the only way to win.

6 piece tablebases say that Rh6 and Rh5 are also ways to win....

[smirobth]

Here is a position that is solved instantly with the R+P vs R tablebase, but no programs solve it without them:

[D]k7/P6R/3K4/8/7P/2r5/8/8 w - - 0 1
Rh8+! is the only way to win.

6 piece tablebases say that Rh6 and Rh5 are also ways to win....

You are correct. I should have said that giving up the a-pawn is the only way to win (and that Rh8+ is the quickest). However without access to 5 man tablebases programs all hang on to the pawn and play moves like h5, which only draws.