michiguel wrote:Hi Wylie,
I confine one pawn to a side, and then I have 4096 king combinations. In other words, I do not discard illegal positions in these situations. The fact is that it does not matter too much because it compresses really well. I think that being too clever at this point it does not pay off because it may makes the files more difficult to compress. OTOH, having simpler indexing scheme helps in calculating them faster. I did not do too much research on this, but the fact is that I ended up with files that are smaller than Nalimov's.
The idea to confine the pawn first is based on the fact that I will have 24 "slices" of that particular file that could be calculated independently. When the pawn is in 7th, I do not need to know any info about positions with the pawn in 6th determine the DTM info with all the combinations of pieces. In other words, I calculate and finish all the info with the pawn in A7, then I could do the same with the pawn in A6 (now it will need the info of positions with the pawn in A7) etc.
That helps a lot because the RAM required to be allocated is 1/24th of the size of the file. That is why GTBs do not require too much RAM at all to calculate TBs with pawns.
Miguel
Makes sense. I don't think 1806/3612 would harm compression but it probably wouldn't help much either. The way it might be useful is taking up less space in RAM when uncompressed, which maybe is not too important.
I've given some thought to slicing up databases according to the pawn indexing terms, so that any pawn move takes you to a new slice. The nice thing about it is that pawn moves take you from slice to slice and reversible moves do not. Enpassant might be a bit tricky though with two or more pawns. I had come up with the idea of processing a "set" of "slices" at once, where the set would contain at least two pawn configurations.... the current one, and the mirror of the current one. White king moves crossing the center of the board would trigger mirroring and cross into the mirrored slice. Only the king move could trigger this, so pawn captures (even enpassant captures) would not change the indexing of the other pieces (i.e. the index within the slice would not change during a pawn move). When a double pawn push was possible, it would take you into an "enpassant slice" from which all of your moves, EP capture or not, would take you back into a normal slice.
Slicing like this might also help with castling rights. Each slice without castling rights would also have zero or more related slices with castling rights -- one or both kings pinned to their square, and one or more rooks removed from the indexing of the slice. So those would be much smaller. But anyway, all irreversible moves (pawn pushes, captures or promotions, and castling) would either move into a different slice or into a different database. All reversible moves would always stay within the same slice, except for white king moves that cross the center and trigger mirroring (or I guess any king move could trigger it, in endings without pawns).
Anyway its great that your databases are available for engines to use and for people to experiment with, thanks again.