7-men Syzygy attempt

[Vinvin]

...
I think it would be fun to have my very own 7 man file set. I have a great big NAS cabinet, and I can just add on 12 TB disks as they get built. I guess that storage costs will continue to fall over time so it may not be as expensive as one might think.

And Intel Optane is coming : https://www.anandtech.com/show/12512/th ... p-review/5

[Dann Corbit]

About the "find_val:not found" problem happened at the third reduction, I think this line may need to be changed into

Code: Select all

j = CAPT_CWIN_RED + 1 + i - REDUCE_PLY_RED;

I'm still not quite sure about this.

Once you get the code fixed, I can do some runs on a high end machine if you like. I just bought a bunch of SSDs including 2x2TB and 5 smaller ones.

I think it would be fun to have my very own 7 man file set. I have a great big NAS cabinet, and I can just add on 12 TB disks as they get built. I guess that storage costs will continue to fall over time so it may not be as expensive as one might think.

Look like you have been solving storage problem for 7 men!!!

How about RAM? Will you soon have 1 TB?

I have two machines with 128 GB, but 1TB RAM is a lot of cash right now

[Dann Corbit]

Concerning DTM bases. Have you checked whether placing them on regular hard disk (HDD) inflict/cause noticeabe performance drop. Similarly like dtz works fine on HDD without noticeable performance penalty.

HDD is fine if they are probed only at the root (although finding the whole winning line can take a while).

Probing DTM during the search really needs them to be on SSD.

Two Terabyte SSD drives can be bought for $360 now. The prices are finally starting to drop again.

Those super-high speed gumstick SSDs can be had for abouit $400 per Terabyte. Probably not necessary for using the SSD files, but I bet they are great for building them.

[Nordlandia]

Maybe PCIe or NVMe types of SSDs is preferable for 7-man syzygy.

I'm worried that regular consumer SSDs cam reduce engine speed if probing 7-man. Probing full 7-man set can inflict moderate speed penalty compared to low/slight speed penalty of 6-man.

Dann Corbit: if Ronald's 500 GB estimation for DTM bases is correct, 1 TB SSD is sufficient enough for syzygy tablebases.

500 + 150 = 650. 350 GB available for something else.

[Dann Corbit]

Maybe PCIe or NVMe types of SSDs is preferable for 7-man syzygy.

I'm worried that regular consumer SSDs cam reduce engine speed if probing 7-man. Probing full 7-man set can inflict moderate speed penalty compared to low/slight speed penalty of 6-man.

Dann Corbit: if Ronald's 500 GB estimation for DTM bases is correct, 1 TB SSD is sufficient enough for syzygy tablebases.

500 + 150 = 650. 350 GB available for something else.

He is working on DTM. If those fit on 500GB for 7 men, I will be more than a little bit surprised. In fact astonished is not enough.

I also guess that the win from a 7 man probe hit is much greater than 6 man, which is already better than 5 man.

In ten years a TB of RAM will be affordable on desktop PCs for the average consumer.

And if the probes cost too much, then reduce the fraction of times that you probe. For instance, probing only at the root is cheap, and probing only at the root and at pv nodes is also cheap.

[Vinvin]

...
I think it would be fun to have my very own 7 man file set. I have a great big NAS cabinet, and I can just add on 12 TB disks as they get built. I guess that storage costs will continue to fall over time so it may not be as expensive as one might think.

And Intel Optane is coming : https://www.anandtech.com/show/12512/th ... p-review/5

I just found a video about new Optane products here :

[noobpwnftw]

I did change the stat limits according to their pawn version like so.

Now MAX_PLY=768 should be sufficient for 7-men TBs, I will verify the generation process works without problem before moving on to the compression part.

[syzygy]

I did change the stat limits according to their pawn version like so.

Now MAX_PLY=768 should be sufficient for 7-men TBs, I will verify the generation process works without problem before moving on to the compression part.

I will add my 16-bit permutation and compression code to github, but I will probably not have time before the weekend.

The DTZ format has to be adapted to allow 16-bit values in the "map"' array. It would be possible to do this just for those tables that need it, so the tables already generated remain valid.

For pawns it might be good to make a bigger change: compressing data per rank instead of per file. That might lead to somewhat better compression, but it is also a bit more complicated and it will need some amount of programming. I have done this for DTM, but in a kind of brute force way (generating the whole table at once, which is not an option for 7 pieces). I guess the generation part can still be done per file.

[Ras]

Btw, in tbcore.c open_tb(), I think there is a buffer overflow waiting to happen when copying over the path names.

[noobpwnftw]

I will add my 16-bit permutation and compression code to github, but I will probably not have time before the weekend.

Thank you, and I think it is possible to differentiate between file formats with the fifth byte in the header or just use another magic value.

With your new 16-bit compression code available, I'm pretty confident that full set of 7-men Syzygy will be available soon.

I already have the necessary hardware(both steps in physical RAM without --disk) and will provide hosting for all the resulting files.