7-men Syzygy attempt

Discussion of chess software programming and technical issues.

Moderators: hgm, Harvey Williamson, bob

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
Vinvin
Posts: 4137
Joined: Thu Mar 09, 2006 8:40 am

Re: 7-men Syzygy attempt

Post by Vinvin » Mon Mar 12, 2018 2:34 am

Dann Corbit wrote:...
I think it would be fun to have my very own 7 man file set. I have a great big NAS cabinet, and I can just add on 12 TB disks as they get built. I guess that storage costs will continue to fall over time so it may not be as expensive as one might think.
And Intel Optane is coming : https://www.anandtech.com/show/12512/th ... p-review/5 8-)

Dann Corbit
Posts: 9051
Joined: Wed Mar 08, 2006 7:57 pm
Location: Redmond, WA USA
Contact:

Re: 7-men Syzygy attempt

Post by Dann Corbit » Mon Mar 12, 2018 6:58 am

phhnguyen wrote:
Dann Corbit wrote:
noobpwnftw wrote:About the "find_val:not found" problem happened at the third reduction, I think this line may need to be changed into

Code: Select all

j = CAPT_CWIN_RED + 1 + i - REDUCE_PLY_RED;
I'm still not quite sure about this.
Once you get the code fixed, I can do some runs on a high end machine if you like. I just bought a bunch of SSDs including 2x2TB and 5 smaller ones.

I think it would be fun to have my very own 7 man file set. I have a great big NAS cabinet, and I can just add on 12 TB disks as they get built. I guess that storage costs will continue to fall over time so it may not be as expensive as one might think.
Look like you have been solving storage problem for 7 men!!!

How about RAM? Will you soon have 1 TB? ;)
I have two machines with 128 GB, but 1TB RAM is a lot of cash right now
Taking ideas is not a vice, it is a virtue. We have another word for this. It is called learning.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.

Dann Corbit
Posts: 9051
Joined: Wed Mar 08, 2006 7:57 pm
Location: Redmond, WA USA
Contact:

Re: 7-men Syzygy attempt

Post by Dann Corbit » Mon Mar 12, 2018 7:12 am

syzygy wrote:
Nordlandia wrote:Concerning DTM bases. Have you checked whether placing them on regular hard disk (HDD) inflict/cause noticeabe performance drop. Similarly like dtz works fine on HDD without noticeable performance penalty.
HDD is fine if they are probed only at the root (although finding the whole winning line can take a while).

Probing DTM during the search really needs them to be on SSD.
Two Terabyte SSD drives can be bought for $360 now. The prices are finally starting to drop again.

Those super-high speed gumstick SSDs can be had for abouit $400 per Terabyte. Probably not necessary for using the SSD files, but I bet they are great for building them.
Taking ideas is not a vice, it is a virtue. We have another word for this. It is called learning.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.

User avatar
Nordlandia
Posts: 2082
Joined: Fri Sep 25, 2015 7:38 pm
Location: Sortland, Norway

Re: 7-men Syzygy attempt

Post by Nordlandia » Mon Mar 12, 2018 9:54 am

Maybe PCIe or NVMe types of SSDs is preferable for 7-man syzygy.

I'm worried that regular consumer SSDs cam reduce engine speed if probing 7-man. Probing full 7-man set can inflict moderate speed penalty compared to low/slight speed penalty of 6-man.

Dann Corbit: if Ronald's 500 GB estimation for DTM bases is correct, 1 TB SSD is sufficient enough for syzygy tablebases.

500 + 150 = 650. 350 GB available for something else.

Dann Corbit
Posts: 9051
Joined: Wed Mar 08, 2006 7:57 pm
Location: Redmond, WA USA
Contact:

Re: 7-men Syzygy attempt

Post by Dann Corbit » Mon Mar 12, 2018 12:51 pm

Nordlandia wrote:Maybe PCIe or NVMe types of SSDs is preferable for 7-man syzygy.

I'm worried that regular consumer SSDs cam reduce engine speed if probing 7-man. Probing full 7-man set can inflict moderate speed penalty compared to low/slight speed penalty of 6-man.

Dann Corbit: if Ronald's 500 GB estimation for DTM bases is correct, 1 TB SSD is sufficient enough for syzygy tablebases.

500 + 150 = 650. 350 GB available for something else.
He is working on DTM. If those fit on 500GB for 7 men, I will be more than a little bit surprised. In fact astonished is not enough.

I also guess that the win from a 7 man probe hit is much greater than 6 man, which is already better than 5 man.

In ten years a TB of RAM will be affordable on desktop PCs for the average consumer.

And if the probes cost too much, then reduce the fraction of times that you probe. For instance, probing only at the root is cheap, and probing only at the root and at pv nodes is also cheap.
Taking ideas is not a vice, it is a virtue. We have another word for this. It is called learning.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.

Vinvin
Posts: 4137
Joined: Thu Mar 09, 2006 8:40 am

Re: 7-men Syzygy attempt

Post by Vinvin » Tue Mar 13, 2018 1:19 pm

Vinvin wrote:
Dann Corbit wrote:...
I think it would be fun to have my very own 7 man file set. I have a great big NAS cabinet, and I can just add on 12 TB disks as they get built. I guess that storage costs will continue to fall over time so it may not be as expensive as one might think.
And Intel Optane is coming : https://www.anandtech.com/show/12512/th ... p-review/5 8-)
I just found a video about new Optane products here : https://www.youtube.com/watch?v=oWqO36Zj65k

noobpwnftw
Posts: 258
Joined: Sun Nov 08, 2015 10:10 pm

Re: 7-men Syzygy attempt

Post by noobpwnftw » Tue Mar 13, 2018 3:42 pm

I did change the stat limits according to their pawn version like so.

Now MAX_PLY=768 should be sufficient for 7-men TBs, I will verify the generation process works without problem before moving on to the compression part.

syzygy
Posts: 4343
Joined: Tue Feb 28, 2012 10:56 pm

Re: 7-men Syzygy attempt

Post by syzygy » Tue Mar 13, 2018 4:10 pm

noobpwnftw wrote:I did change the stat limits according to their pawn version like so.

Now MAX_PLY=768 should be sufficient for 7-men TBs, I will verify the generation process works without problem before moving on to the compression part.
I will add my 16-bit permutation and compression code to github, but I will probably not have time before the weekend.

The DTZ format has to be adapted to allow 16-bit values in the "map"' array. It would be possible to do this just for those tables that need it, so the tables already generated remain valid.

For pawns it might be good to make a bigger change: compressing data per rank instead of per file. That might lead to somewhat better compression, but it is also a bit more complicated and it will need some amount of programming. I have done this for DTM, but in a kind of brute force way (generating the whole table at once, which is not an option for 7 pieces). I guess the generation part can still be done per file.

Ras
Posts: 1018
Joined: Tue Aug 30, 2016 6:19 pm
Contact:

Re: 7-men Syzygy attempt

Post by Ras » Tue Mar 13, 2018 5:17 pm

Btw, in tbcore.c open_tb(), I think there is a buffer overflow waiting to happen when copying over the path names.

noobpwnftw
Posts: 258
Joined: Sun Nov 08, 2015 10:10 pm

Re: 7-men Syzygy attempt

Post by noobpwnftw » Tue Mar 13, 2018 5:25 pm

I will add my 16-bit permutation and compression code to github, but I will probably not have time before the weekend.
Thank you, and I think it is possible to differentiate between file formats with the fifth byte in the header or just use another magic value.

With your new 16-bit compression code available, I'm pretty confident that full set of 7-men Syzygy will be available soon.

I already have the necessary hardware(both steps in physical RAM without --disk) and will provide hosting for all the resulting files.

Post Reply