How is work on 8-man tablebases progressing?

[towforce]

If there is any genuine determination behind what people say, then there should already be an independently verified 7-piece TB result.

So far I have not seen it. Things are often easier said than done, at least Ronald, me and few others got something done.

IIRC, the 7-man tablebase was built in 2018. There's no proof that it's 100% correct, but I have no recollection of anybody finding any errors in it either. That tells me that it's pretty damn good - which is good enough for me.

IMO it's value is greater than helping chess engines (which, to me personally, is of no value at all - the chess.com bots are more than good enough for my needs): it's nice to be able to look positions up and get the actual solution.

[Jouni]

I think 8-man can give 0-5 elo gain like 7-man. No practical benefit. Only academic interest. I don't use even 6-man. If there is interesting position I check DTM from Shredderchess .

[chesskobra]

Have they tried to implement something like this: we compute all 8 piece positions (so definitely computational resource is needed), but we store a minimal subset of the positions such that anything that is not in the subset can be reduced to a position in the subset 'on the fly' (in a few seconds on a reasonable hardware). As our computers become more powerful, we will need to store smaller subsets. How much saving of diskspace can we hope to achieve this way? Such an idea may be tried on 3-4-5 piece databases.

[Koistinen]

The best incentive ever to solve chess: save the planet from having to produce millions of unnecessary chips and many gigawatts of electricity just to get to 8-man!!!

With my simple inefficient scheme 690,330,986,839,277,568 bytes would be needed to store all 8-man tablebases.
If you assume each computer connected to the net would draw 1kW and have 20TB of disk and you need 3x as much storage for a distributed solution to make it work ok, you still only need 104k computers and so only 104MW of power. This is not even a single GW.
We need to do all 9-man to need 15GW.

[syzygy]

I don't know how many people would be interested in spending enormous resources to calculate tables which are merely approximately correct.

But they would not be 'approximately correct'. They would be merely incomplete. By handicapping the strong side with some restrictions, like not being allowed to give black the opportunity to move a promoted Pawn, only draws would be suspect. And many of the Pawn slices would have no draw positions at all (other than those where the draw is reached through immediate conversion).

Why not simply not calculate the 8-piece tables at all?

Silly question, which you could ask for anything that is discussed on this forum. Why not refrain from any programming activity in the area of Chess? Choose live!

But sad as it might be, people like their computers to play strong Chess, and they think that end-game tables can improve the quality of their analysis, or at least speed it up. So they would like to have tables, even if these are just DTZ or WDL, and that includes 8-man tables. And they won't get those by 'simply not calculating them at all'...

Well, at least I don't see any point in spending half the effort to do something half.

If there is a particular position you want to solve, and you know that a trick can get you the answer more quickly, then great. But the point of calculating 8-men TBs is to calculate them. Like calculating a trillion digits of pi, no point in accepting a solution that you know is only 99.999999% right.

[syzygy]

Have they tried to implement something like this: we compute all 8 piece positions (so definitely computational resource is needed), but we store a minimal subset of the positions such that anything that is not in the subset can be reduced to a position in the subset 'on the fly' (in a few seconds on a reasonable hardware). As our computers become more powerful, we will need to store smaller subsets. How much saving of diskspace can we hope to achieve this way? Such an idea may be tried on 3-4-5 piece databases.

Current compression schemes already throw away a lot of information that can be quickly recalculated. For example, my DTZ tables only store the smaller of the white-to-move and black-to-move tables, and the WDL tables "don't store" positions with a winning capture. (The position is stored, but the value encoded is a "don't care" value picked for good compression. Drawn positions with a drawing capture are encoded as either loss or draw -> the probing code will figure out that it is a draw even if it is encoded as a loss.) Removing a whole side-to-move from WDL would result in major slow downs during search (and during generation of other tables with pawns that promote into your table).

To "throw away" more positions, you have to figure out a concrete, precise, and 100% logically correct scheme that allows you to reconstruct the removed values for those positions in a very short amount of time. This is not easy.

Another idea is to come up with a predictor function and store the delta between the predicted values and the calculated values. But you would have to do this for each table, and if the predictors take 1ms per position, you are looking at 1296 YEARS of computation time spent only on calculating prediction values for a single pawnless 8-piece table (without double pieces).

[hgm]

Well, at least I don't see any point in spending half the effort to do something half.

That is already a really weird attitude; half the job done without introducing any loss of efficiency should be considered a good start.

It becomes an even more counter-productive stance when you apply it to doing, say, 99% of the job for 1% of the effort. Especially if the remaining 1% concerns only cases that no one is interested in.

Your pi metaphor sucks. You should have said "no point in calculating the first 12 digits of pi when you cannot calculate the next million digits". Engineers would not agree...

[towforce]

Well, at least I don't see any point in spending half the effort to do something half.

That is already a really weird attitude; half the job done without introducing any loss of efficiency should be considered a good start.

It becomes an even more counter-productive stance when you apply it to doing, say, 99% of the job for 1% of the effort. Especially if the remaining 1% concerns only cases that no one is interested in.

Your pi metaphor sucks. You should have said "no point in calculating the first 12 digits of pi when you cannot calculate the next million digits". Engineers would not agree...

Person 1: "We sent a crew to the moon, and people are very impressed!"

Person 2: "Excellent! Now let's send a crew to Mars!"

Person 1: We can't afford to send a crew to Mars.

Person 2: OK then - let's send a crew a third of the way to Mars, and then bring them back!


Not convinced that you'd get voters backing the idea of spending a huge sum of money to go a third of the way to Mars and then come back.

[hgm]

So what is there 1/3 of the way to Mars worth going to?

Person 1: "Let's send a crew to the Mars, people will be impressed".
Person 2: "Could you also send a crew to Jupiter?".
Person 1: "No, that is too expensive, and there is nothing but gas there!".
Person 2: "Then I wouldn't waste any money going to Mars".

Or:
Person 1: "It's only two years to the end of the decade. Let's send a crew to orbit the Moon!"
Person 2: "Will they be able to land on it?"
Person 1: "No, the moonlander isn't ready yet."
Person 2: "Then there is no point, no one will be interested."

[Koistinen]

By the time that storage requirements for 8-piece tables are manageable, requiring 64TB of RAM will probably be acceptable. (And for most tables this number can be reduced by quite a bit by rewriting the generator without completely abandoning the approach of "in RAM generation". I wrote my generator just to do 6-piece tables on a decent PC, and for that task taking into account "like-piece symmetry" wasn't worth the trouble. For 7-piece and 8-piece this is different.)

I computed the size requirements without and with reducing using permutations of same pieces to illustrate.
Note that the number for reducing is just an approximation, dividing by the number of permutations of like pieces for those endgames.
Last on each line is the percentage of space you need if you don't use like-piece symmetry.

3: 524288 524288 100
4: 306184192 290979840 105
5: 96921976832 79135331668 122
6: 22360673484800 14305093222384 156
7: 4218203345518592 1947456033067182 216
8: 690330986839277568 213529294167604807 323
9: 101523728307603898368 19652448336691703337 516
10: 13728689646745625296896 1561065282681230985761 879
11: 1734484955477896291418112 109176580993481410741492 1588
12: 207127250281663267673735168 6824378818319881564216564 3035