Gaviota tablebases probing code (v0.3) UPDATE

[michiguel]

http://sites.google.com/site/gaviotache ... e/releases

1) Multiple paths can be added in one line separated by ';'
2) tb_init() does not output info to stdout when verbosity=1. Now it returns a string (a pointer to an internal buffer) in which the initialization info is contained. In this way, this info could be saved to a log file, send to stdout etc. The internal buffer could be static or dynamic. This is not a concern for the developer of the engine. If the buffer is dynamic, it will be free() at the end when tb_done() is executed. Currently, the buffer is static, but it may change in the future. It is guaranteed that this buffer will only change when tb_init(), tb_restart(), or tb_done() are executed.

An example of this info string is:

Code: Select all

GTB PATHS
  main: gtb/gtb4
    #1: gtb/gtb3
    #2: gtb/gtb2
    #3: gtb/gtb1

GTB initialization
  Compression  Scheme = 4
  Compression Indexes (3-pc) = PASSED
  Compression Indexes (4-pc) = **FAILED**
  Compression Indexes (5-pc) = **FAILED**

3) tb_availability() has been implemented. It return information about what type of files are available. 3, 4 or 5-men and whether they are complete or not. It is useful to auto-detect what is available.
4) Paths passed to path_add() are not needed to be unmodified until tbpaths_done(). Now they are copied internally.
5) Several compiler warnings have been silenced
6) readme.txt file has been modified

This modification will not break your current code. If you chose not to return anything when using tb_init(), that is legal C code.

Miguel

[Aaron Becker]

Thanks for the update, Miguel. It's very convenient to let your library handle the path splitting, and more diagnostic information is always nice.

[wgarvin]

Just curious...

From looking at the probing code, in endings without pawns you index according to the 462 legal king combinations with 8-fold symmetry, but it looks as if you didn't do the same in endings with pawns (1806 combinations with 2-fold symmetry)?

You could still confine one pawn to a side, if you wanted--then you'd enumerate 3612 king combinations. I think that is what the Darkthought team did. I think it would be simpler to confine the white king though, because of endings with more than one pawn (and especially because of endings that can have EP captures).

Either way, the savings is not as dramatic as the 462 term but it would still reduce the index space by about 11.8% when pages are uncompressed and cached.

[Edit: by the way, great work getting all of this going... hopefully your etb's will become the new standard and lots of engines will benefit from your effort!]

[michiguel]

Just curious...

From looking at the probing code, in endings without pawns you index according to the 462 legal king combinations with 8-fold symmetry, but it looks as if you didn't do the same in endings with pawns (1806 combinations with 2-fold symmetry)?

You could still confine one pawn to a side, if you wanted--then you'd enumerate 3612 king combinations.

Hi Wylie,

I confine one pawn to a side, and then I have 4096 king combinations. In other words, I do not discard illegal positions in these situations. The fact is that it does not matter too much because it compresses really well. I think that being too clever at this point it does not pay off because it may makes the files more difficult to compress. OTOH, having simpler indexing scheme helps in calculating them faster. I did not do too much research on this, but the fact is that I ended up with files that are smaller than Nalimov's.

The idea to confine the pawn first is based on the fact that I will have 24 "slices" of that particular file that could be calculated independently. When the pawn is in 7th, I do not need to know any info about positions with the pawn in 6th determine the DTM info with all the combinations of pieces. In other words, I calculate and finish all the info with the pawn in A7, then I could do the same with the pawn in A6 (now it will need the info of positions with the pawn in A7) etc.

That helps a lot because the RAM required to be allocated is 1/24th of the size of the file. That is why GTBs do not require too much RAM at all to calculate TBs with pawns.

Miguel

I think that is what the Darkthought team did. I think it would be simpler to confine the white king though, because of endings with more than one pawn (and especially because of endings that can have EP captures).

Either way, the savings is not as dramatic as the 462 term but it would still reduce the index space by about 11.8% when pages are uncompressed and cached.

[Edit: by the way, great work getting all of this going... hopefully your etb's will become the new standard and lots of engines will benefit from your effort!]

[wgarvin]

Hi Wylie,

I confine one pawn to a side, and then I have 4096 king combinations. In other words, I do not discard illegal positions in these situations. The fact is that it does not matter too much because it compresses really well. I think that being too clever at this point it does not pay off because it may makes the files more difficult to compress. OTOH, having simpler indexing scheme helps in calculating them faster. I did not do too much research on this, but the fact is that I ended up with files that are smaller than Nalimov's.

The idea to confine the pawn first is based on the fact that I will have 24 "slices" of that particular file that could be calculated independently. When the pawn is in 7th, I do not need to know any info about positions with the pawn in 6th determine the DTM info with all the combinations of pieces. In other words, I calculate and finish all the info with the pawn in A7, then I could do the same with the pawn in A6 (now it will need the info of positions with the pawn in A7) etc.

That helps a lot because the RAM required to be allocated is 1/24th of the size of the file. That is why GTBs do not require too much RAM at all to calculate TBs with pawns.

Miguel

Makes sense. I don't think 1806/3612 would harm compression but it probably wouldn't help much either. The way it might be useful is taking up less space in RAM when uncompressed, which maybe is not too important.

I've given some thought to slicing up databases according to the pawn indexing terms, so that any pawn move takes you to a new slice. The nice thing about it is that pawn moves take you from slice to slice and reversible moves do not. Enpassant might be a bit tricky though with two or more pawns. I had come up with the idea of processing a "set" of "slices" at once, where the set would contain at least two pawn configurations.... the current one, and the mirror of the current one. White king moves crossing the center of the board would trigger mirroring and cross into the mirrored slice. Only the king move could trigger this, so pawn captures (even enpassant captures) would not change the indexing of the other pieces (i.e. the index within the slice would not change during a pawn move). When a double pawn push was possible, it would take you into an "enpassant slice" from which all of your moves, EP capture or not, would take you back into a normal slice.

Slicing like this might also help with castling rights. Each slice without castling rights would also have zero or more related slices with castling rights -- one or both kings pinned to their square, and one or more rooks removed from the indexing of the slice. So those would be much smaller. But anyway, all irreversible moves (pawn pushes, captures or promotions, and castling) would either move into a different slice or into a different database. All reversible moves would always stay within the same slice, except for white king moves that cross the center and trigger mirroring (or I guess any king move could trigger it, in endings without pawns).

Anyway its great that your databases are available for engines to use and for people to experiment with, thanks again.