7-men Syzygy attempt

[Dann Corbit]

As of now, it is highly likely to indicate wdl is to be less or more than 10 TB in size. For simple adjudication only wdl is needed. 10 TB HDDs is fairly affordable.

There is a commercial 100 TB SSD drive.
I guess in 5 years a 100 TB SSD drive will be less than $1000.
Anyone who wants will be able to afford storage for the full 7 man files.

7 man files is around the corner. Storing them in SSD format is extremely expensive. Alternatively using HDD for adjudication in cutechess is less expensive. In the long run like you're implying, more and more people can afford them for raw analysis. HDD simply inflict severe bottleneck, thus only choise is for adjudication.

Currently, you can get a 2TB SSD for about $350.
In 5 years 100 TB will be less than a thousand dollars.
If you want 7 man tablebase files, it will be affordable then (at least to some).

There is no bottleneck. Even on slow HDD, it is all about how you probe them.

And adjudication won't work unless you have them stored somewhere. So the same thing applies to TB files used for adjudication.

100 TB SSD:
http://nimbusdata.com/?gclid=Cj0KCQjwkp ... TUEALw_wcB

Prices fall exponentially over time. Capacity rises exponentially over time.
Eventually, it will cost one penny to store 10 man TB files.
Maybe that is a very long time from now. But eventually.

In the meantime, 7 man TB will be cheap enough for nut case hobbyists like me very soon.

[noobpwnftw]

So "Skylape Scalable Performance" is not so scalable?

Digging into the specs and various tests shows that newer generation divided the memory controlling unit and made them local to cores, resulting lower per-core bandwidth but higher combined, same to LLC, this gives performance improvements under normal lock-free cases but I guess in our lock heavy scenario it would then need to signal all other cores about of the cache eviction and thus causing a bus contention. Manual also said that even a "pause" on Skylake sometimes need more than 200 cycles as opposed to the previous generations, probably due to the same reason.

Also the slowdown of NUMA remote node is almost by a half because it is on a 8-way system. But I have some good results running with 64 threads, pqos tool(on-chip performance counter) shows almost full memory bandwidth used and no backfire.

Optimising NUMA memory allocation together with thread binding might still help a bit. I will have a look at it.

Usually it can be done with invoking generator via "numactl --interleave=all". But depending on different hardware setup it is hard to decide an optimal distribution. Linux kernel has NUMA balancers and thread migration which suppose to do such things automagically.

Btw, are there differences in scalability between the generator the permutator and the compressor? It might make sense to use different numbers of threads for these parts...

It only happens with the generator, specially(calc_mate, iteration), where there are more locks.

[syzygy]

But calc_mates() does not use locked instructions, so if it is slow it is because of the large amount of random memory accesses.

I think calc_mates() can actually be sped up a lot, but I have to figure out if what I am doing in my DTM generator would also work in the WDL/DTZ generator.

[syzygy]

I think calc_mates() can actually be sped up a lot, but I have to figure out if what I am doing in my DTM generator would also work in the WDL/DTZ generator.

It seems to be more a simplification than a speed up.

Instead of verifying that an in-check position is MATE in calc_mates(), the position could be set to CHANGE. The iter() loop can then be used to verify the mate just as it is used to verify losses with DTZ > 0. This means that the special /* MATE */ case in iter() could go (and also check_mate()). However, check_mate() seems more efficient than the more general check_loss(), so it is not clear that this would be a win.

[Morten Lohne]

Currently, you can get a 2TB SSD for about $350.
In 5 years 100 TB will be less than a thousand dollars.

SSDs have become at most 4x cheaper in the last 5 years. More like 3x, really. You're delusional if you think they'll suddenly get 20x cheaper by 2023.

[Dann Corbit]

Currently, you can get a 2TB SSD for about $350.
In 5 years 100 TB will be less than a thousand dollars.

SSDs have become at most 4x cheaper in the last 5 years. More like 3x, really. You're delusional if you think they'll suddenly get 20x cheaper by 2023.

There was a local bubble in memory prices.
It's going away now. Micro trends are just noise on the curve.

You're delusional if you think it's not going to happen.

[noobpwnftw]

I think calc_mates() can actually be sped up a lot, but I have to figure out if what I am doing in my DTM generator would also work in the WDL/DTZ generator.

It seems to be more a simplification than a speed up.

Instead of verifying that an in-check position is MATE in calc_mates(), the position could be set to CHANGE. The iter() loop can then be used to verify the mate just as it is used to verify losses with DTZ > 0. This means that the special /* MATE */ case in iter() could go (and also check_mate()). However, check_mate() seems more efficient than the more general check_loss(), so it is not clear that this would be a win.

It is more than likely that anyone who's building 7 pieces would have more cycles than memory bandwidth, if things did not go south I think the difference is minor. It normally took about 5 minutes for me on the calc_mates step per table.

[Ozymandias]

Prices fall exponentially over time. Capacity rises exponentially over time.

This was true, more or less (I haven't tried to fit the data to a curve), until 7 years ago; since then, price per gigabyte remains basically flat and higher capacity drives show increases that are marginal, at best. At least in Europe.

[koedem]

Cool, I just looked at the stats of the most recent one ftp://112.73.74.24/pub/syzygy/7men_testing/KQQRNvKQ.txt and found it interesting that there actually are cursed wins for white. So black has 50 checks but yet at some point runs out.
Also Black can have wins up to 21 plies. Is there any good way to find these interesting positions in the TB file?
EDIT: I'm dumb, he listed those exact positions in the last lines. Thanks a lot.

[Milos]

There was a local bubble in memory prices.
It's going away now. Micro trends are just noise on the curve.

You're delusional if you think it's not going to happen.

Sorry Dann, but it's you who is delusional. You have to first understand technology and its limits to see what will happen with prices.
NAND flash is currently going from TLC to QLC which at best is 25% increase in capacity but in reality it is less because of increased overprovisioning.
We are now just starting with 128 layers, compared to 64L the price per GB didn't drop yet at all and in 2 years time might drop 20%.
In 5 years we'll be at 256L and at best with further 20% decrease.
So if you look overall price per GB might drop by 50% in an optimistic scenario by 2023, but most probably it'll drop much less.
So even if we reach 1000TB discs for consumer market (which is another SF, we won't even have 50TB discs by 2023) the price would be over 100k$.