Syzygy tablebases

[syzygy]

For under 1K you can buy dual E5-2670 with 1TB of RAM. 3 such machines are much faster and better choice for generating 7-men than 44 cores E5-2699-v4 machine with 1.5TB or RAM.
P.S. Is is only 88 threads, but 44 real cores. You know E5-2xxx in Xeon marking comes from dual, meaning you can put only 2 of those on a board.

I can't reproduce a 1K dual E5-2670 with 1 Tb of RAM. The two 2670s are under $200 each, but where do you get such cheap RAM? Note that the E5-2670 is limited to DDR3-1600 RAM, for which goes for $760 per 64 Gb (2 x 32 Gb) at Newegg. That's $6K for "only" 0.5 Tb and total build costs near $8K. Or are you willing to buy used sticks? Also, the SuperMicro X9D series of motherboards do not support more than 0.5 Tb anyway.

The E5-2670 itself does not support more than 384GB, so a dual E5-2670 will be stuck at 768GB. My stupid generator would need a bit more than 924 GB.

768GB will mean 24 x 32 GB sticks. Cheap mainboards with 24 slots are probably not easy to find and cheap 32 GB DDR3 sticks even less so. Ebay does not know about RAM modules with more than 8 GB.

32GB DDR4 sticks are cheaper than DDR3 sticks, which might be sufficient reason to forget about the older hardware generations when it comes to generating 7-piece TBs.

Unfortunately the price of 32GB DDR4 sticks has been increasing considerably over the last months. And of course to get to 1 TB 32GB will not suffice, you'll need 16 x 64GB DDR4 LRDIMMs, which seem to go for about € 700 and 16 x € 700 = € 11,200. Still, about 25K would seem to suffice for a system with dual E5-2699v4s, 1 TB RAM and some big HDDs.

But then... who is going to use those tablebases? Having them on HDD is only a bit more useful than probing the Lomonosov tables online. You need the WDL tables on SSD for them to be really useful (and ideally you should have a few hundred GB for caching them in RAM). The Samsung 850 EVO 4TB SSD costs about € 1,350. Two of them are probably not yet sufficient, but about € 3,000 seems to be a realistic cost for storing the 7-piece WDL tables. And then there's the problem of downloading / distributing 10-20 TB of data.

[Evert]

How much can you save by only considering interesting cases?

What I mean by that is that if you know how to play, say KQkn, then it is not so interesting to consider KQXkn. The only interesting tables are KQknx.

[syzygy]

How much can you save by only considering interesting cases?

What I mean by that is that if you know how to play, say KQkn, then it is not so interesting to consider KQXkn. The only interesting tables are KQknx.

If a WDL table is completely won for one side, it will compress to almost nothing for that reason.

[Vinvin]

But then... who is going to use those tablebases? Having them on HDD is only a bit more useful than probing the Lomonosov tables online. You need the WDL tables on SSD for them to be really useful (and ideally you should have a few hundred GB for caching them in RAM). The Samsung 850 EVO 4TB SSD costs about € 1,350. Two of them are probably not yet sufficient, but about € 3,000 seems to be a realistic cost for storing the 7-piece WDL tables. And then there's the problem of downloading / distributing 10-20 TB of data.

The point is to get the most interesting 7pcs TBs.
And the most interesting is KRPPKRP.

[APassionForCriminalJustic]

But then... who is going to use those tablebases? Having them on HDD is only a bit more useful than probing the Lomonosov tables online. You need the WDL tables on SSD for them to be really useful (and ideally you should have a few hundred GB for caching them in RAM). The Samsung 850 EVO 4TB SSD costs about € 1,350. Two of them are probably not yet sufficient, but about € 3,000 seems to be a realistic cost for storing the 7-piece WDL tables. And then there's the problem of downloading / distributing 10-20 TB of data.

The point is to get the most interesting 7pcs TBs.
And the most interesting is KRPPKRP.

Yes why can't we just create the 7-men that would be most common, and complex. I think that the kings, pawns, plus rooks - and kings, pawns, plus queens would be invaluable.

[syzygy]

But then... who is going to use those tablebases? Having them on HDD is only a bit more useful than probing the Lomonosov tables online. You need the WDL tables on SSD for them to be really useful (and ideally you should have a few hundred GB for caching them in RAM). The Samsung 850 EVO 4TB SSD costs about € 1,350. Two of them are probably not yet sufficient, but about € 3,000 seems to be a realistic cost for storing the 7-piece WDL tables. And then there's the problem of downloading / distributing 10-20 TB of data.

The point is to get the most interesting 7pcs TBs.
And the most interesting is KRPPKRP.

Yes why can't we just create the 7-men that would be most common, and complex. I think that the kings, pawns, plus rooks - and kings, pawns, plus queens would be invaluable.

Go ahead.

[APassionForCriminalJustic]

But then... who is going to use those tablebases? Having them on HDD is only a bit more useful than probing the Lomonosov tables online. You need the WDL tables on SSD for them to be really useful (and ideally you should have a few hundred GB for caching them in RAM). The Samsung 850 EVO 4TB SSD costs about € 1,350. Two of them are probably not yet sufficient, but about € 3,000 seems to be a realistic cost for storing the 7-piece WDL tables. And then there's the problem of downloading / distributing 10-20 TB of data.

The point is to get the most interesting 7pcs TBs.
And the most interesting is KRPPKRP.

Yes why can't we just create the 7-men that would be most common, and complex. I think that the kings, pawns, plus rooks - and kings, pawns, plus queens would be invaluable.

Go ahead.

Your current probing code wouldn't work for 7-men though right? And how would you generate them? I know with Nalimov that one can pick and choose which tables that they would like to have. So I only downloaded endgames that are most likely to be useful for an engine during probing. That's why it would be cool if even just some 7-men could be created so that engines such as Stockfish could use them. But how big would even just say a KRPPKRP be?

[Rein Halbersma]

But how big would even just say a KRPPKRP be?

Simple: 462 * choose(62,2) * choose(60,2) * 57 = 88 billion positions, so with WLD (2 bits per position) that's 22 Gb of disk, uncompressed. With compression that will be many times smaller. The problem with 7 pc tables is the sheer number of different tables, not as much the size per table.

[Milos]

For under 1K you can buy dual E5-2670 with 1TB of RAM. 3 such machines are much faster and better choice for generating 7-men than 44 cores E5-2699-v4 machine with 1.5TB or RAM.
P.S. Is is only 88 threads, but 44 real cores. You know E5-2xxx in Xeon marking comes from dual, meaning you can put only 2 of those on a board.

I can't reproduce a 1K dual E5-2670 with 1 Tb of RAM. The two 2670s are under $200 each, but where do you get such cheap RAM? Note that the E5-2670 is limited to DDR3-1600 RAM, for which goes for $760 per 64 Gb (2 x 32 Gb) at Newegg. That's $6K for "only" 0.5 Tb and total build costs near $8K. Or are you willing to buy used sticks? Also, the SuperMicro X9D series of motherboards do not support more than 0.5 Tb anyway.

The E5-2670 itself does not support more than 384GB, so a dual E5-2670 will be stuck at 768GB. My stupid generator would need a bit more than 924 GB.

768GB will mean 24 x 32 GB sticks. Cheap mainboards with 24 slots are probably not easy to find and cheap 32 GB DDR3 sticks even less so. Ebay does not know about RAM modules with more than 8 GB.

32GB DDR4 sticks are cheaper than DDR3 sticks, which might be sufficient reason to forget about the older hardware generations when it comes to generating 7-piece TBs.

Unfortunately the price of 32GB DDR4 sticks has been increasing considerably over the last months. And of course to get to 1 TB 32GB will not suffice, you'll need 16 x 64GB DDR4 LRDIMMs, which seem to go for about € 700 and 16 x € 700 = € 11,200. Still, about 25K would seem to suffice for a system with dual E5-2699v4s, 1 TB RAM and some big HDDs.

But then... who is going to use those tablebases? Having them on HDD is only a bit more useful than probing the Lomonosov tables online. You need the WDL tables on SSD for them to be really useful (and ideally you should have a few hundred GB for caching them in RAM). The Samsung 850 EVO 4TB SSD costs about € 1,350. Two of them are probably not yet sufficient, but about € 3,000 seems to be a realistic cost for storing the 7-piece WDL tables. And then there's the problem of downloading / distributing 10-20 TB of data.

SUPERMICRO X9DRi-F is around 400$ on Amazon and supports up to 1TB of LRDIMM (X9DR7-TF+ with 24 slots is around 700$). Problem is 64 or even 32GB modules, since you can't find them so easily on ebay especially for reasonable price. 16GB modules you can find already for 4$ per GB.
In terms of processor power for computer chess Broadwell is total waste of money.
If you run SF8 benchmark on a single core of E5-2699v4 running on 3.6GHz, you'll get exactly the same NPS as if you run SF8 benchmark on single core of AMD Phenom II X6 1100T OCed at 4.2GHz (and that one is not even bmi2 capable). Meaning 7 years older AMD is only 15% slower per core on equal frequency than the most state-of-the-art Intel Xeon (if you run non-bmi2 compile you lose even that advantage)!

[kbhearn]

The other trouble with wanting just the interesting tables is often they have the uninteresting tables as dependencies.

i.e. for a simple generator KRPP v KRP depends on all KRxx v KRx tables being generated first, however uncommon they are in practice and however unlikely they are to impact the table you actually want

You could try to build a clever generator that escapes some of those dependencies but doing so correctly is not trivial.