New tablebase probing/interface code (and probing tool)

[syzygy]

Same here, so there is a bug somewhere...

Fixed:

Code: Select all

$ ./jprobetool "8/8/3P4/4K3/4P1k1/5q2/8/8 b - - 0 1"
Found 97 WDL, 97 DTZ and 0 DTM tablebase files.

 +---+---+---+---+---+---+---+---+
 |   |   |   |   |   |   |   |   | 8
 +---+---+---+---+---+---+---+---+
 |   |   |   |   |   |   |   |   | 7
 +---+---+---+---+---+---+---+---+
 |   |   |   | P |   |   |   |   | 6
 +---+---+---+---+---+---+---+---+
 |   |   |   |   | K |   |   |   | 5
 +---+---+---+---+---+---+---+---+
 |   |   |   |   | P |   | k |   | 4
 +---+---+---+---+---+---+---+---+
 |   |   |   |   |   | q |   |   | 3
 +---+---+---+---+---+---+---+---+
 |   |   |   |   |   |   |   |   | 2
 +---+---+---+---+---+---+---+---+
 |   |   |   |   |   |   |   |   | 1
 +---+---+---+---+---+---+---+---+
   a   b   c   d   e   f   g   h

WDL = -2 (loss)
DTZ = -2 (ply)

DTZ-optimal mating line:
[FEN "8/8/3P4/4K3/4P1k1/5q2/8/8 b - - 0 1"]
1...Kg3 2.d7 Kf2 3.Kd4 Ke1 4.d8Q Kd1 5.e5 Kc1 6.e6 Kb1 7.e7 Ka1 8.e8Q Kb1 9.Ke3
Qg2 10.Kf2 Qh1 11.Kg1 Ka1 12.Kxh1 wins by baring the king

With 2 or more pawns, the "leading pawn" for the index calculation has to be move to the front of a list. A small simplification I made in tbprobe.c/jtbprobe.c caused it to be moved back again.

So this does not affect the validity of the tables.

[syzygy]

Code: Select all

DTZ-optimal mating line:
[FEN "8/8/3P4/4K3/4P1k1/5q2/8/8 b - - 0 1"]
1...Kg3 2.d7 Kf2 3.Kd4 Ke1 4.d8Q Kd1 5.e5 Kc1 6.e6 Kb1 7.e7 Ka1 8.e8Q Kb1 9.Ke3
Qg2 10.Kf2 Qh1 11.Kg1 Ka1 12.Kxh1 wins by baring the king

And this seems to be a good example of ugly DTZ play. White just looks for the quickest zeroing move that preserves the win, which are pawn moves, and black cannot stop them, so just lets its king wander into the direction of a1 because of the order in which moves are generated.

[sscg13]

Prolix on that position with tablebase gives something like
info depth 38 nodes 64788876 tbhits 53051671 time 18563 score cp -25000 wdl 0 0 1000 pv f3g2 d6d7 g2f3 e5d5 g4f4 e4e5 f4f5 d5d6 f5e4 e5e6 e4f5 e6e7 f5e4 e7e8q f3e2 e8f7 e4d4 d7d8q e2f3 d8c7 f3e2 c7b6 d4e4 f7g6 e2d3 d6e6 d3c4 g6f5 e4d4 e6d6 c4d3 b6c5 d4c4 f5e6 d3e2 c5b6 c4d4
which is rather similar to the non-TB pv and does not blunder a trivial mate

A quick test on a balanced book resulted in the version with WDL probing getting 4 wins, 3 losses, 193 draws (which probably speaks more to the fact that these are identical engines for most of the game) but at the very least there were no crashes. I'll probably add root probing soon now that I'm reasonably confident everything else works.

[syzygy]

Prolix on that position with tablebase gives something like
info depth 38 nodes 64788876 tbhits 53051671 time 18563 score cp -25000 wdl 0 0 1000 pv f3g2 d6d7 g2f3 e5d5 g4f4 e4e5 f4f5 d5d6 f5e4 e5e6 e4f5 e6e7 f5e4 e7e8q f3e2 e8f7 e4d4 d7d8q e2f3 d8c7 f3e2 c7b6 d4e4 f7g6 e2d3 d6e6 d3c4 g6f5 e4d4 e6d6 c4d3 b6c5 d4c4 f5e6 d3e2 c5b6 c4d4
which is rather similar to the non-TB pv and does not blunder a trivial mate

Does this mean you are letting your engine probe in the eval and not in the search() function itself?

[sscg13]

I keep probing to search function, and only start returning scores when the number of pieces is less than at root. This means it should prefer finding a material simplification, but most winning KXX v KX require both pieces, so it does not often lead to strange play. Otherwise, I use WDL probing to prune moves that do not keep the result. The reason why it appears a TB score is being returned there is because I apply a correction at root when printing the UCI info lines.

[syzygy]

I keep probing to search function, and only start returning scores when the number of pieces is less than at root. This means it should prefer finding a material simplification, but most winning KXX v KX require both pieces, so it does not often lead to strange play. Otherwise, I use WDL probing to prune moves that do not keep the result. The reason why it appears a TB score is being returned there is because I apply a correction at root when printing the UCI info lines.

OK! That is an excellent approach!

[phhnguyen]

I have just created this new repository:
https://github.com/syzygy1/probetool

I wrote the code such that it should be easy to integrate the probing code into an engine or GUI.

Thank you a lot for your excellent work!
Could you write clearly about the license of this code, please? I guess what it is, but I still prefer a clear thing.

Of course Fathom already exists:
https://github.com/jdart1/Fathom
And there is probably no valid reason to switch if your engine uses Fathom.
(Unless your engine wants to learn to play Shatranj, for which I am going to write probing code next.)

A user of my chess GUI reported that it took too much memory when running with Syzygy 7 men for a while. My chess GUI uses a modified Fathom. I can't check since I don't have Syzygy 7. Could you manage to check/test both Fathom and your new code with Syzygy 7 men? Thanks a lot again.

[syzygy]

I have just created this new repository:
https://github.com/syzygy1/probetool

I wrote the code such that it should be easy to integrate the probing code into an engine or GUI.

Thank you a lot for your excellent work!
Could you write clearly about the license of this code, please? I guess what it is, but I still prefer a clear thing.

There are no restrictions or requirements on the use of the code.
No permission is needed to use, modify and distribute it all you want.
You can even pick a license and pretend it is released under that license.

Of course Fathom already exists:
https://github.com/jdart1/Fathom
And there is probably no valid reason to switch if your engine uses Fathom.
(Unless your engine wants to learn to play Shatranj, for which I am going to write probing code next.)

A user of my chess GUI reported that it took too much memory when running with Syzygy 7 men for a while. My chess GUI uses a modified Fathom. I can't check since I don't have Syzygy 7. Could you manage to check/test both Fathom and your new code with Syzygy 7 men? Thanks a lot again.

I don't have 7 men either. But what you can do, certainly in a GUI, is call TB_release() to unmmap() the TB files.

Code: Select all

// TB_release() will release any mmap()ed tablebase files, reducing memory
// usage. This could be called between games, for example.
void TB_release(void);

When you call TB_release() there should not be any ongoing TB probes. Once TB_release() returns, you can continue probing.

In Fathom a similar effect can probably be achieved by calling tb_init(""), i.e. with empty path, which releases everything, and then calling tb_init() again with the correct path, which rescans the TB paths (which TB_release() avoids). But I did not check this.

Someone using 7-men tables should have enough RAM that there is no need for a swap file. So the swap file should be disabled -> no swapping! (and the OS will be forced instead to release RAM used to cache mmap()d TB files, which is exactly what you want).