New tablebase probing/interface code (and probing tool)

[syzygy]

I have just created this new repository:
https://github.com/syzygy1/probetool

probetool is a simple cli tool for probing tablebase positions specified by a FEN:

Code: Select all

$ ./probetool "K7/N7/k7/8/3p4/8/N7/8 w - -"
Found 163 WDL, 163 DTZ and 0 DTM tablebase files.

 +---+---+---+---+---+---+---+---+
 | K |   |   |   |   |   |   |   | 8
 +---+---+---+---+---+---+---+---+
 | N |   |   |   |   |   |   |   | 7
 +---+---+---+---+---+---+---+---+
 | k |   |   |   |   |   |   |   | 6
 +---+---+---+---+---+---+---+---+
 |   |   |   |   |   |   |   |   | 5
 +---+---+---+---+---+---+---+---+
 |   |   |   | p |   |   |   |   | 4
 +---+---+---+---+---+---+---+---+
 |   |   |   |   |   |   |   |   | 3
 +---+---+---+---+---+---+---+---+
 | N |   |   |   |   |   |   |   | 2
 +---+---+---+---+---+---+---+---+
 |   |   |   |   |   |   |   |   | 1
 +---+---+---+---+---+---+---+---+
   a   b   c   d   e   f   g   h

WDL = 1 (cursed win)
DTZ50 = 164 ply

DTZ50-optimal mating line:
[FEN "K7/N7/k7/8/3p4/8/N7/8 w - -"]
1.Nb4+ Kb6 2.Nd3 Kc7 3.Nb5+ Kc6 4.Na3 Kb6 5.Kb8 Kc6 6.Nc2 Kb5 7.Kc7 Ka4 8.Kb6
Kb3 9.Nce1 Ka2 10.Kc5 Kb3 11.Kb5 Kc3 12.Ka4 Kd2 13.Kb3 Kd1 14.Nf3 Ke2 15.Nfe5
Kd2 16.Kb2 Kd1 17.Nc4 Ke2 18.Kc2 Kf3 19.Kd1 Ke4 20.Ke2 Kd5 21.Nd2 Kc6 22.Kf3
Kd5 23.Kf4 Ke6 24.Kg5 Kd5 25.Kf5 Kd6 26.Ke4 Ke6 27.Nc4 Kf6 28.Kd5 Ke7 29.Ke5
Kf7 30.Kd6 Kf6 31.Nd2 Kf5 32.Ke7 Kg6 33.Ke6 Kg5 34.Ke5 Kg6 35.Ne4 Kg7 36.Kd6
Kh6 37.Nd2 Kg7 38.Ke6 Kf8 39.Nc4 Ke8 40.Na3 Kd8 41.Nb5 Kc8 42.Kd6 Kb7 43.Kd7
Ka6 44.Na3 Ka5 45.Nb1 Kb5 46.Nd2 Kb6 47.Kd6 Kb5 48.Kc7 Ka6 49.Kc6 Ka5 50.Kc5
Ka6 51.Nc4 Kb7 52.Kd6 Kc8 53.Na5 Kd8 54.Nb7+ Ke8 55.Ke6 Kf8 56.Nd6 Kg7 57.Kf5
Kh6 58.Kf6 Kh5 59.Ne4 Kg4 60.Ng5 Kh4 61.Kf5 Kg3 62.Ke4 Kg4 63.Nf7 Kg3 64.Nfe5
Kh4 65.Kf5 Kh5 66.Ng4 Kh4 67.Nf6 Kh3 68.Ke5 Kg3 69.Ke4 Kh3 70.Kf3 Kh4 71.Kf4
Kh3 72.Ne4 Kh4 73.Ng3 Kh3 74.Nf5 Kg2 75.Kg4 Kf1 76.Ng3+ Kg1 77.Ne4 Kh2 78.Nd2
Kg2 79.Kh4 Kg1 80.Kh3 Kh1 81.Kg3 Kg1 82.Nf2 d3 83.Nd1 Kh1 84.Nf3 d2 85.Nf2#

I wrote the code such that it should be easy to integrate the probing code into an engine or GUI.

Of course Fathom already exists:
https://github.com/jdart1/Fathom
And there is probably no valid reason to switch if your engine uses Fathom.
(Unless your engine wants to learn to play Shatranj, for which I am going to write probing code next.)

[Joerg Oster]

I have just created this new repository:
https://github.com/syzygy1/probetool

probetool is a simple cli tool for probing tablebase positions specified by a FEN:

Code: Select all

$ ./probetool "K7/N7/k7/8/3p4/8/N7/8 w - -"
Found 163 WDL, 163 DTZ and 0 DTM tablebase files.

 +---+---+---+---+---+---+---+---+
 | K |   |   |   |   |   |   |   | 8
 +---+---+---+---+---+---+---+---+
 | N |   |   |   |   |   |   |   | 7
 +---+---+---+---+---+---+---+---+
 | k |   |   |   |   |   |   |   | 6
 +---+---+---+---+---+---+---+---+
 |   |   |   |   |   |   |   |   | 5
 +---+---+---+---+---+---+---+---+
 |   |   |   | p |   |   |   |   | 4
 +---+---+---+---+---+---+---+---+
 |   |   |   |   |   |   |   |   | 3
 +---+---+---+---+---+---+---+---+
 | N |   |   |   |   |   |   |   | 2
 +---+---+---+---+---+---+---+---+
 |   |   |   |   |   |   |   |   | 1
 +---+---+---+---+---+---+---+---+
   a   b   c   d   e   f   g   h

WDL = 1 (cursed win)
DTZ50 = 164 ply

DTZ50-optimal mating line:
[FEN "K7/N7/k7/8/3p4/8/N7/8 w - -"]
1.Nb4+ Kb6 2.Nd3 Kc7 3.Nb5+ Kc6 4.Na3 Kb6 5.Kb8 Kc6 6.Nc2 Kb5 7.Kc7 Ka4 8.Kb6
Kb3 9.Nce1 Ka2 10.Kc5 Kb3 11.Kb5 Kc3 12.Ka4 Kd2 13.Kb3 Kd1 14.Nf3 Ke2 15.Nfe5
Kd2 16.Kb2 Kd1 17.Nc4 Ke2 18.Kc2 Kf3 19.Kd1 Ke4 20.Ke2 Kd5 21.Nd2 Kc6 22.Kf3
Kd5 23.Kf4 Ke6 24.Kg5 Kd5 25.Kf5 Kd6 26.Ke4 Ke6 27.Nc4 Kf6 28.Kd5 Ke7 29.Ke5
Kf7 30.Kd6 Kf6 31.Nd2 Kf5 32.Ke7 Kg6 33.Ke6 Kg5 34.Ke5 Kg6 35.Ne4 Kg7 36.Kd6
Kh6 37.Nd2 Kg7 38.Ke6 Kf8 39.Nc4 Ke8 40.Na3 Kd8 41.Nb5 Kc8 42.Kd6 Kb7 43.Kd7
Ka6 44.Na3 Ka5 45.Nb1 Kb5 46.Nd2 Kb6 47.Kd6 Kb5 48.Kc7 Ka6 49.Kc6 Ka5 50.Kc5
Ka6 51.Nc4 Kb7 52.Kd6 Kc8 53.Na5 Kd8 54.Nb7+ Ke8 55.Ke6 Kf8 56.Nd6 Kg7 57.Kf5
Kh6 58.Kf6 Kh5 59.Ne4 Kg4 60.Ng5 Kh4 61.Kf5 Kg3 62.Ke4 Kg4 63.Nf7 Kg3 64.Nfe5
Kh4 65.Kf5 Kh5 66.Ng4 Kh4 67.Nf6 Kh3 68.Ke5 Kg3 69.Ke4 Kh3 70.Kf3 Kh4 71.Kf4
Kh3 72.Ne4 Kh4 73.Ng3 Kh3 74.Nf5 Kg2 75.Kg4 Kf1 76.Ng3+ Kg1 77.Ne4 Kh2 78.Nd2
Kg2 79.Kh4 Kg1 80.Kh3 Kh1 81.Kg3 Kg1 82.Nf2 d3 83.Nd1 Kh1 84.Nf3 d2 85.Nf2#

I wrote the code such that it should be easy to integrate the probing code into an engine or GUI.

Of course Fathom already exists:
https://github.com/jdart1/Fathom
And there is probably no valid reason to switch if your engine uses Fathom.
(Unless your engine wants to learn to play Shatranj, for which I am going to write probing code next.)

Thank you!

[sscg13]

I actually do have a fairly strong (around 3500 CCRL Blitz 1CPU or 3600 8CPU equivalent) shatranj engine (at https://github.com/sscg13/Prolix). Earlier I tried modifying Fathom and I can't say it really worked out, so I will try using your probing and see how it goes.

[syzygy]

I actually do have a fairly strong (around 3500 CCRL Blitz 1CPU or 3600 8CPU equivalent) shatranj engine (at https://github.com/sscg13/Prolix). Earlier I tried modifying Fathom and I can't say it really worked out, so I will try using your probing and see how it goes.

Interesting! Let me know if you encounter any problems.

[Dave Gomboc]

I have just created this new repository:
https://github.com/syzygy1/probetool

Of course Fathom already exists:
https://github.com/jdart1/Fathom
And there is probably no valid reason to switch if your engine uses Fathom.
(Unless your engine wants to learn to play Shatranj, for which I am going to write probing code next.)

Interesting.

1. Could performance perhaps be one valid reason? Do you believe that this new probing code is likely to execute significantly faster than the probing code in Jon Dart's Fathom repository (https://github.com/jdart1/fathom/) and/or the modified Fathom available within https://github.com/davegomboc/egt_comparison?

2. Is this probing tool fully usable in the complete absence of a chess engine? For instance, how does this new Syzygy probing tool handle provided FENs that represent:
a) a position where the player to move has just been checkmated
b) a position where the player to move has just been stalemated
c) a position where the player not to move is in check
d) a position where each player does not have exactly one king
et cetera.

[syzygy]

I have just created this new repository:
https://github.com/syzygy1/probetool

Of course Fathom already exists:
https://github.com/jdart1/Fathom
And there is probably no valid reason to switch if your engine uses Fathom.
(Unless your engine wants to learn to play Shatranj, for which I am going to write probing code next.)

Interesting.

1. Could performance perhaps be one valid reason? Do you believe that this new probing code is likely to execute significantly faster than the probing code in Jon Dart's Fathom repository (https://github.com/jdart1/fathom/) and/or the modified Fathom available within https://github.com/davegomboc/egt_comparison?

Perhaps, but Fathom doesn't seem to do things very differently, and I can't really tell which will be faster.
If you need to construct the bitboards passed to Fathom by scanning your board array, then Fathom might be at a disadvantage.
But it depends how you use my code. Interface directly, or via tbinterface_bb/_0x88 and write a fast conversion routine to initialize a TB_Position struct, or pass the position in a FEN string (not recommended for engines).

I used to think that the best way to interface with the probing code was to use your own engine's move generation and board representation to do the TB-internal capture resolution, but it might actually be better to use something much simpler like I have now implemented in tbinterface_bb.c and tbinterface_0x88.c. Especially if your regular do_move() automatically updates NNUE data structures etc.

If an engine already uses bitboard-based move generation, then one could modify tbinterface_bb.c to use the engine's bitboard routines and avoid having duplicate magic or pext tables.

2. Is this probing tool fully usable in the complete absence of a chess engine? For instance, how does this new Syzygy probing tool handle provided FENs that represent:
a) a position where the player to move has just been checkmated
b) a position where the player to move has just been stalemated
c) a position where the player not to move is in check
d) a position where each player does not have exactly one king
et cetera.

It should reject positions in c) and d) (and positions with castling rights, and positions with too many pieces). It should report the correct values for a) and b).

But interestingly it accepts a position with two white kings, so there is a bug (fixed now).

[sscg13]

I actually do have a fairly strong (around 3500 CCRL Blitz 1CPU or 3600 8CPU equivalent) shatranj engine (at https://github.com/sscg13/Prolix). Earlier I tried modifying Fathom and I can't say it really worked out, so I will try using your probing and see how it goes.

Interesting! Let me know if you encounter any problems.

Would there be any issue with moving the declaration of TB_Position to jtbinterface.h? Without this, the compiler complains about incomplete type usage.

[syzygy]

I actually do have a fairly strong (around 3500 CCRL Blitz 1CPU or 3600 8CPU equivalent) shatranj engine (at https://github.com/sscg13/Prolix). Earlier I tried modifying Fathom and I can't say it really worked out, so I will try using your probing and see how it goes.

Interesting! Let me know if you encounter any problems.

Would there be any issue with moving the declaration of TB_Position to jtbinterface.h? Without this, the compiler complains about incomplete type usage.

Yes that's fine.
I kept it as abstract as possible, so probetool (and tbprobe.c) only a see a pointer to TB_Position and not what's in there, but if you want to interface efficiently with it, it is better to include its definition so you can initialize it directly from your own board representation.

Also feel free to remove the "j"s .

[sscg13]

I have mostly worked out WDL probing, though still trying to think of ways to improve the "feel" of play (especially when root position is a draw, is there any better method than just playing a random drawing move?)

In the meanwhile, the probing tool seems to fail on the fen 8/8/3P4/4K3/4P1k1/5q2/8/8 b - - 0 1 which is rather strange to me, I'm not sure if there is a bug? It seems to result in a memory access violation in decompress_pairs; could you take a look into that?

[syzygy]

I have mostly worked out WDL probing, though still trying to think of ways to improve the "feel" of play (especially when root position is a draw, is there any better method than just playing a random drawing move?)

When the root is draw, you can do a regular search (but without TB probing) on all moves that draw.

If you use the root_probe_wdl() function (or root_probe_dtz() if DTZ is available) in jprobetool.c, you will get a score and a rank for each move in RootMoves[]. The idea of the rank is that you do a regular search on the set of moves with the highest rank.

The moves in RootMove[] are meaningless integers, but you can convert them to uci move notation using TBitf_move_to_string_uci().
Or you modify root_probe_wdl/dtz() to use your own move generator for the main loop.

In the meanwhile, the probing tool seems to fail on the fen 8/8/3P4/4K3/4P1k1/5q2/8/8 b - - 0 1 which is rather strange to me, I'm not sure if there is a bug? It seems to result in a memory access violation in decompress_pairs; could you take a look into that?

Same here, so there is a bug somewhere...