Syzygy bases from memory

[Rebel]

Nah, draws can be cut off as a real transposition, bit-bases can't, you missed that and snp it.

What are you babbling about? I think you have lost everyone now. But perhaps that is the entire idea.

I think the entire idea is to lure me into one of your fights, won't happen, but keep on trying

[hgm]

OK, so your idea is that this is a fight. That explains a lot.

I, however, are interested in chessprogramming facts. And it should be clear to all readers now that for perfect evaluation of draws in end-games that have on the order of 10% (or 90%), by far the best method is just store the draw/non-draw bitbase in memory, and probe the relevant bit by a simple memory access. Other methods are of course possible, but they require many times the amount of memory, and are easily an order of magnitude slower.

Especially just storing a table of only the draws (or wins) is not competative, because 10% is just not few enough to earn back the position encoding this requires.

Don't be fooled by persons who claim the contrary and are evasive when asked for details or throw smoke. They are wrong, and they know it.

[Rebel]

The endgame neural nets in SlowChess are primarily for board positions larger than 5-man usually multiple pieces and any number of pawns. They are trained on W/L/D result of the sample games so that is similar (though eventually started blending in some search scoring after initial training on some of them.) I do think training a neural net to replace a tablebase could do pretty well for a much smaller size, probably well enough that the difference would be hard to measure. But can't beat perfect, I'm sure I'd end up seeing some loss like maybe in TCEC if I dropped tablebases then want to re-add support.

I plan on trying to use distance-to-mate in the target value to train a mating net (for no-pawns positions) to replace my hard-coded MatingEval function, to see if it can find mates quicker, just as an experiment. But probably won't try specific endgames with only a few pieces.

The SlowChess bitbase for KP-KR is 428 kb, it only stores a win/draw bit. I would have to look to see how else it's stored, my bitbases are from 2005 I think haha. I think the syzygy for that endgame are actually smaller though? I was surprised at how small the syzygy were when I first looked at them.

I remember for some like KPKQ I had some piece position checks and only stored possibly drawn positions so that one is only 40kb.

I looked into your code, saw it came under the UBC license (Use But Credit) and wondered if I could use it, did some test positions and it did very well, is it 100% accurate? But my compiler could not compile it and I gave up. Regarding KPKQ, I prune too, white can only draw if the pawn is at least at the 6th row and the white king is close to the pawn, and if the black queen is not on the promotion square.

[Rebel]

OK, so your idea is that this is a fight. That explains a lot.

I, however, are interested in chessprogramming facts. And it should be clear to all readers now that for perfect evaluation of draws in end-games that have on the order of 10% (or 90%), by far the best method is just store the draw/non-draw bitbase in memory, and probe the relevant bit by a simple memory access. Other methods are of course possible, but they require many times the amount of memory, and are easily an order of magnitude slower.

Especially just storing a table of only the draws (or wins) is not competative, because 10% is just not few enough to earn back the position encoding this requires.

Don't be fooled by persons who claim the contrary and are evasive when asked for details or throw smoke. They are wrong, and they know it.

I am not in the mood to rehash my goals, Chris kindly remembered you, you choose to be deaf. The choice for a TT is because a) I don't need bit-based for my goal and b) the TT approach outperforms bit-bases because I can use it in search as well.

Now, you may have the last word, be kind for a change.

[hgm]

I would like to be 'kind', but if you write nonsense, it is a tad hard to remain truthful and call it anything but nonsense... This is a chessprogramming forum with many readers, so it is important that info exhibited here is factually correct and not misleading. And if you consider it unkind to be caught at an error, then so be it.

For one, what 'TT approach'? In usual chessprogramming jargon TT means Transposition Table, and it is the first time you mention it in this respect. (And not clear how it would be useful in this context either.) In a previous posting you said that you 'hashed' the drawn positions (presumably meaning converting the position to a unique number, not necessarily storing it in a hash table), and using a binary search to find back the key (similar to how Polyglot implements its opening book).

As I and several others pointed out: this is bulky (because the keys are many bits, not 1) and slow (because binary search needs log(N) ~ 20 probes into the bulky set that doesn't fit in cache, rather than a single probe into a data set so small it might even fit in L3). Yet you keep insisting it is faster...

It is a complete mystery why you now suddenly claim bitbases cannot be used in search. Especially since this is what everyone who uses bitbases does. And it makes no sense: a bitbase probe gives you exactly the same information as the binary search, namely whether a given position is a draw or not. So of course you can use it for the same purpose, whatever that purpose may be. It is just nonsensical to claim otherwise.

[Rebel]

Ah, you could not, not a real surprise.

Once again and last -

I DON'T NEED BIT-BASES

Now shut up and leave me alone.

[hgm]

It is not relevant what you need or don't need. The world doesn't revolve around you. What is important is that the readers can learn the best way to solve the indicated problem here. If you want to show them how better not to do that, that's fine too.

[Rebel]

It's not a matter what is best, you judge and you have no right. I am perfectly fine with the chosen format as it contains the best move for free to guide the search something bit-bases don't have. No loss in NPS, a binary search in memory is peanuts. Within a couple of days I will release the positions of KPK, KRKP, KQKP, KRKN and KRKB with Syzygy score and best move in EPD format. Anyone interested can create their own chosen format, whatever that suits them best. There is no absolute truth in this matter, you claim it, utter nonsense.

[syzygy]

I want to fix some holes in my static KPvK evaluation from the 80's. It catches most (80-90%) of the winning positions but not all. Installing the Syzygy bases is of course the most simple solution but it is slow because it runs from HD. The question is if it can run from memory and if loading "KPvK.rtbw" and "KPvK.rtbz" at program start is sufficient.

It should all be in the system cache in RAM after the first probe and stay there. It's not slow (especially if you are just probing through it once to find the holes).

[syzygy]

(If I really wanted my TB's in memory, both wouldn't be an issue for me, because this system has 32GB RAM and boots in under a minute as it is now. The most time-consuming part of the process is actually booting the UEFI and having it run its checks.)

If you want the TBs to be in RAM during probing, forget about RAM drives. They just take memory away from the RAM available for the system cache. (Unless you have a smart RAM drive implementation, but I doubt those exist on Windows.)