Syzygy bases from memory

[Rebel]

[d]8/K7/1p6/8/8/8/k7/2R5 b - - bm b5; ce 0; acd 1;
[d]8/r7/7K/8/8/8/5P2/2k5 w - - bm f4; ce 0; acd 1;
Takes ProDeo 20 plies to see the draw (SF=17), with a KRP table one ply and it operates through the whole tree without NPS loss.

[hgm]

Of course this is partly due to the fact that the 'default' evaluation of KRKP is wrong. KRKP is either an easy win or a draw. But if it is a win, the path to a winning conversion is usually short, while in case of a draw you usually have to stay pretty long in it, until promotion forces conversion to KK (postponing that with checks as long as it can). So it is better to evaluate it as a draw by default. Then you will always get the correct score after just a few moves. If you evaluate it as a win (i.e. +4) by default, it will take a large depth before you can guarantee a correct result.

[Rebel]

For KRKP (36 million positions) my only interest are draw scores, only those (3.8 million) go into a the table else the table becomes too big. And since these 3.8 million are checked with the Syzygy bases the result is always correct. For KPK / KRKN / KRKB my only interest are the won positions, only those go into the table, another case is the KQKP ending with a pawn on the 6/7th row, some of them are draws -> into the table. As for the path to win or draw simply add a small PST value.

[hgm]

I am not sure what exactly you are doing, but it sounds like you have a self-inflicted problem here, by using an inferior storage format compared to the EGT. If you are only interested in draw positions, an EGT would require only a single bit per position (1=draw, 0=other). But that only works for a complete EGT, where the position is implied by the index. If you want to store 'only the draws', you will have to encode the position.

How do you get to 36M positions anyway? 64^4 is only 16M, if you want to tabulate for both stm that makes 32M, including all illegal and broken positions, and Pawns on 1st/8th rank. But encoding 32M positions by a unique number requires 25 bits. Encoding 10% of the positions (the draws) as a 25-bit number takes 2.5 times as much space as the raw, uncompressed (!) EGT, which uses only 1 bit per position.

[Rebel]

I am aware of the 4-6 man bit-bases, but I don't want them all, I dont care about KQQK etc. I just want the most common my HCE engine doesn't fully understand ending up with something like this:

[hgm]

Well, then you only use EGT for these. The point is that the most compact form to store the info you want is as EGT. Even uncompressed EGTs are far smaller than any other form of tabulation. For maximum speed you probably would want uncompressed, but even then a draw/non-draw 4-men EGT with Pawns would only require 1.5MB.

[chrisw]

For KRKP (36 million positions) my only interest are draw scores, only those (3.8 million) go into a the table else the table becomes too big. And since these 3.8 million are checked with the Syzygy bases the result is always correct. For KPK / KRKN / KRKB my only interest are the won positions, only those go into the table, another case is the KQKP ending with a pawn on the 6/7th row, some of them are draws -> into the table. As for the path to win or draw simply add a small PST value.

My mind is wondering off here into a sorted list of hash indices for the win positions and then a binary search into the list to see if the position is in it. I guess you wouldn’t need hash indices, just a position index would do (so few pieces). Or am I just over complicating it?

[hgm]

Just blowing up size by a factor 5 or so. And access time by a factor 20. How many bits do you think you would need to identify a 4-men position?

[mar]

For KRKP (36 million positions) my only interest are draw scores, only those (3.8 million) go into a the table else the table becomes too big. And since these 3.8 million are checked with the Syzygy bases the result is always correct. For KPK / KRKN / KRKB my only interest are the won positions, only those go into the table, another case is the KQKP ending with a pawn on the 6/7th row, some of them are draws -> into the table. As for the path to win or draw simply add a small PST value.

My mind is wondering off here into a sorted list of hash indices for the win positions and then a binary search into the list to see if the position is in it. I guess you wouldn’t need hash indices, just a position index would do (so few pieces). Or am I just over complicating it?

I've been thinking for a while that I too would like to know whether a position is a draw only, and that's 1 bit per index, however when I counted how many indices I'd need I quickly realized then even this is a huge amount of data

for KRKP I count this: 32*64*64*48*2 = 12 582 912 indices, this is still 32-bit (if I understand your idea), I thought of this too if a vast majority is winning/losing, then you'd only need to store indices for drawn positions
however, that still inflates by a factor of 32, so you'd need less than 1/32 drawn positions (or the opposite if the majority is draws) to actually save space (which I don't believe is the case here) plus I bet indices (even delta coded) would compress worse, would have to be tried though, perhaps byte slicing would help

[chrisw]

For KRKP (36 million positions) my only interest are draw scores, only those (3.8 million) go into a the table else the table becomes too big. And since these 3.8 million are checked with the Syzygy bases the result is always correct. For KPK / KRKN / KRKB my only interest are the won positions, only those go into the table, another case is the KQKP ending with a pawn on the 6/7th row, some of them are draws -> into the table. As for the path to win or draw simply add a small PST value.

My mind is wondering off here into a sorted list of hash indices for the win positions and then a binary search into the list to see if the position is in it. I guess you wouldn’t need hash indices, just a position index would do (so few pieces). Or am I just over complicating it?

I've been thinking for a while that I too would like to know whether a position is a draw only, and that's 1 bit per index, however when I counted how many indices I'd need I quickly realized then even this is a huge amount of data

for KRKP I count this: 32*64*64*48*2 = 12 582 912 indices, this is still 32-bit (if I understand your idea), I thought of this too if a vast majority is winning/losing, then you'd only need to store indices for drawn positions
however, that still inflates by a factor of 32, so you'd need less than 1/32 drawn positions (or the opposite if the majority is draws) to actually save space (which I don't believe is the case here) plus I bet indices (even delta coded) would compress worse, would have to be tried though, perhaps byte slicing would help

I suppose one could view it as a problem of finding the one bits in a (very) sparsely populated bit stream. For (very) simple endings, there'll be quite some periodicity in the stream, you could probably construct the stream using variables like stm and distance to queen and so on. Or, why not reverse the problem, brute force find the variables and the polynomial connecting them backwards off the bit stream. Probably PyTorch or Scipy will do it for you. I jest, alhough something may be possible. Then put that in your evaluation and smoke it.