Tablebase suggestion

[Nordlandia]

How feasible is undertaking 8-man tables or partly about let's say 2030 around. Is the equipment up to snuff, or do we need something like Lomonosov II SuperComputer contraption or deemed obsolete by 2030 to commence 8-man EGTBs generation.

[syzygy]

How feasible is undertaking 8-man tables or partly about let's say 2030 around. Is the equipment up to snuff, or do we need something like Lomonosov II SuperComputer contraption or deemed obsolete by 2030 to commence 8-man EGTBs generation.

For the largest pawnless tables a PC with at least 37.2 TB of shared RAM would be needed if the algorithm uses 1 byte per position.
Most pawnless 8-piece tables will have two identical pieces of the same colour, which would halve the requirements. But 18.6 TB of RAM is still a lot.

It is possible to generate them with less RAM and instead a lot of disk I/O. I think only hard disks could be used for this because SSDs would not survive for long. Generation would be quite slow, but perhaps it is doable.

Pawnful tables could be generated with about 4TB of RAM with 1 byte per position (by splitting the table in 24 parts, pawn on a2-d7) and a reasonable amount of disk I/O. If we don't care about the price (at the moment RAM prices are ridiculously high), then this is possible with a dual-socket or even single-socket EPYC server. But first the relevant pawnless tables would need to be generated.

Total storage for WDL+DTZ should be 1-2 PB or something of that order. This is a lot but it fits in a room. (Whether they can be accessed fast enough for generating tables that promote into them is another question.)

I guess all 8-piece tables will eventually be generated. This is less clear for 9-piece tables.

Here are statistics of tables generated by Marc Bourzutschky:
https://drive.google.com/drive/folders/ ... PuGCkwXDTI
As far as I can tell, all tables have at least two same pieces of the same colour. One of them is KQRBNvKQQ:

Code: Select all

Ending kqrbnkqq
WTM max win:  114
BTM max loss: 114
WTM wins:   3,179,192,378,285 (70.9%)
BTM loses:  1,518,211,034,482 (32.5%)
White wins: 4,697,403,412,767 (51.3%)
WTM legal:  4,483,788,773,672
BTM legal:  4,669,054,760,916

BTM stalemated: 0
WTM winning captures: 2,604,087,801,230
WTM win percent without captures: 30.60
BTM saving captures: 2,710,869,905,387
BTM loss percent without captures: 77.53

Depth          Wins
    1 2683624174292
    2  231127732226
    3  108520251790
    4   65418752434
    ...

I read somewhere that Marc's machine has 1.5 TB of RAM, so his(/Yakov Konoval's) generator might be able to generate the biggest tables using 3 TB of RAM. I don't know if they saved them.

Ah, there is also a readme.txt:

[syzygy]

For the largest pawnless tables a PC with at least 37.2 TB of shared RAM would be needed if the algorithm uses 1 byte per position.

The Wu-Beal algorithm apparently requires only 1 bit per position and only for one side to move, which would mean 2.32 TB.

Once there are two identical pieces, this goes down to 1.16 TB, which indeed fits in 1.5 TB.

So a machine with 4 TB of shared RAM should be enough.
Most pawnless tables will have to be generated using the Wu-Beal algorithm.
Pawnful tables, of which there are many more, can be generated more efficiently, but they are also bigger, and they require a lot of probing of other 8-piece table to resolve pawn promotions, whicih will be slow. (Doing the pawnless tables, which for a large part already seems to have been done, will be the easy part.)

[Nordlandia]

Is it difficult to predict whether it is more difficult to hold on to the 8-piece constellation or whether liquidation to 7-pieces is generally more common?

It has been suggested that eight-piece configurations possess reduced utility, as they can be more quickly liquidated into seven-piece positions.

[syzygy]

Is it difficult to predict whether it is more difficult to hold on to the 8-piece constellation or whether liquidation to 7-pieces is generally more common?

It has been suggested that eight-piece configurations possess reduced utility, as they can be more quickly liquidated into seven-piece positions.

If you pick a position at random, then the more pieces on the board, the higher the probability that the side to move's best move is a capture gaining material. But in chess games pieces generally defend each other, so the interesting positions are not randomly selected.

A better argument "against" the utility of 8-piece tablebases might be that 8-piece positions occurring in games tend to be either too balanced to not be an easy draw, or too unbalanced to not be an easy win for the stronger side. Positions where the material differs by exactly one pawn will always have an odd number of pieces. But of course there are lots of ways in which a 4v4 can be unbalanced, and 5v3 can still be nearly balanced.

[syzygy]

Pawnful tables could be generated with about 4TB of RAM with 1 byte per position (by splitting the table in 24 parts, pawn on a2-d7) and a reasonable amount of disk I/O. If we don't care about the price (at the moment RAM prices are ridiculously high), then this is possible with a dual-socket or even single-socket EPYC server. But first the relevant pawnless tables would need to be generated.

Unfortunately this number applies only to tables with at least 2 pawns (and with two pawns you can split the table into still smaller parts).
Tables with exactly one pawn and no identical pieces need 5.07 TB of RAM. So those would need a different algorithm.
I suppose it should be relatively easy to first generate wtm with 1 byte per position for wtm and 1 bit per position for btm, and then generate btm. This would need 2.85 TB of RAM.

[hgm]

It is possible to exploit the small footprint of the King moves to generate tables with less RAM, swapping to disk without excessive i/o cost. You split the table into K-slices, and at any time you only have a few adjacent K-slices in memory, to treat all moves within and between them. Then you swap one of the K-slices with another on disk, to do the remaining King moves from and to the K-slices that stayed. With cleever indexing the disk access can be largely sequential, and store the data in highly compressed form, as it is usually dealing with sparse bitsets.

[Ender8pieces]

It is possible to exploit the small footprint of the King moves to generate tables with less RAM, swapping to disk without excessive i/o cost. You split the table into K-slices, and at any time you only have a few adjacent K-slices in memory, to treat all moves within and between them. Then you swap one of the K-slices with another on disk, to do the remaining King moves from and to the K-slices that stayed. With cleever indexing the disk access can be largely sequential, and store the data in highly compressed form, as it is usually dealing with sparse bitsets.

I think this is an excellent idea and it could definitely work. I’ve started analyzing different implementation strategies for this.

If we temporarily set aside symmetry (assuming kings can move freely across the entire board) and divide the general slice into 8 parts based on the White King's rank(I think "Batches" is a more appropriate term here than "Slices"), we could operate using the following sliding window logic:

We calculate and write to the first batch (K1) while keeping K2 in memory for reading.
Then, we write to K2 and fetch K3 from storage. Next, we process K3, fetch K4 and evict K1 back to storage, and so on.
This way, we only need to keep roughly 3/8 of the slice in RAM at any given time.

However, adding symmetry into the mix makes it a bit more complex. A Black King's move from rank 4 to 5 triggers a board reflection. In terms of data access, this acts as a "proxy move" for the White King, effectively jumping him from rank 1 to 8, or 2 to 7, etc. This creates non-linear dependencies between the beginning and the end of the file, meaning our "batches" are no longer just sequential but rather linked in a ring-like topology.

I’ve run some initial numbers on this—it increases the I/O overhead slightly, but the memory savings from symmetry still make it the superior approach. I will write down the stages. Assume black king is symmetry on only 10 squares while white king dan be every where.

Write | Fetch | Evict
1 | 1,2,8 |
2 | 3,7 | 8
3 | 4,6 | 1,7
4 | 5 | 2
5 | - |-
6 | 7 | 4
7 | 8,2 | 5
8 | 1 | -

In this method we keep symmetry, read 12 batches instead 8 but holds only 4 batchs in RAM instead 8.
We can further analyze 2D for white king,4D for both kings and even 5D for additional turn separation

[jp]

...
So this is from a game? Which game?

No. I was trying to compose a problem that featured [...]

It is needless to say that I am not a composer, but I like natural looking positions when I try. The arisen position was perfect to test the 8-man online EGTB, which became a subtopic in this thread.

Thanks for clarifying.

...
A better argument "against" the utility of 8-piece tablebases might be that 8-piece positions occurring in games tend to be either too balanced to not be an easy draw, or too unbalanced to not be an easy win for the stronger side. Positions where the material differs by exactly one pawn will always have an odd number of pieces. But of course there are lots of ways in which a 4v4 can be unbalanced, and 5v3 can still be nearly balanced.

Yes, since I read the Op1-TB news in this thread, a number of times I've seen an interesting position... and counted that it has 9 pieces.