Syzygy / egbb discussion

[Ryan Benitez]

why I should invest an extra 80gb of space.

Compared to?

Existing bitbases solutions.

[mvk]

Just tested Toga II v3.0 with Scorpio EGBBs (cache 128M), at 15+0.15 TC, LOS 99.9% stopping:

Code: Select all

    Program                              Score     %     Elo    +   -    Draws

  1 Toga Scorpio                   : 1931.0/3736  51.7     6    8   8   45.0 %
  2 Toga                           : 1805.0/3736  48.3    -6    8   8   45.0 %

12 +/- 8 Elo points benefit for Scorpio EGBBs. By now I tested Nalimov TBs (0+/-2), Shreddder EGBBs (14+/-8), Scorpio EGBBs (12+/-8) and Syzygy TBs (15+/-8). Within error margins, well implemented EGBBs give the same benefit as Syzygy TBs (at least to 3-4-5 men).

The main point for me of Syzygy bases is that they have 6 men WDL and that they are available now. All those tests where the Syzygy's are lobotomized by throwing out the 6 men don't mean too much except confirming that they don't suck. Besides, it is nice that they can be generated within a few days on a relatively simple computer, have a liberal license and unencumbered support. If you don't want the DTZ data, then just remove that and use heuristics to convert a win. (But I don't yet fully understand how pure WDL tables can convert KBNK into a mate. So for the time being I keep them. Disk space is cheap anyway).

[Daniel Shawul]

Just tested Toga II v3.0 with Scorpio EGBBs (cache 128M), at 15+0.15 TC, LOS 99.9% stopping:

Code: Select all

    Program                              Score     %     Elo    +   -    Draws

  1 Toga Scorpio                   : 1931.0/3736  51.7     6    8   8   45.0 %
  2 Toga                           : 1805.0/3736  48.3    -6    8   8   45.0 %

12 +/- 8 Elo points benefit for Scorpio EGBBs. By now I tested Nalimov TBs (0+/-2), Shreddder EGBBs (14+/-8), Scorpio EGBBs (12+/-8) and Syzygy TBs (15+/-8). Within error margins, well implemented EGBBs give the same benefit as Syzygy TBs (at least to 3-4-5 men).

The main point of Syzygy bases is that they have 6 men WDL and that they are available now. All those tests where the Syzygy's are lobotomized by throwing out the 6 men don't mean too much except confirming that they don't suck. Besides, it is nice that they can be generated within a few days on a relatively simple computer, have a liberal license and unencumbered support. If you don't want the DTZ data, then just remove them and use heuristics to convert a win. (But I don't yet fully understand how pure WDL tables can convert KBNK into a mate. So for the time being I keep them. Disk space is cheap anyway).

Well shredder has 40GB bitbases! I could also have generated them 3 years ago since i had the code to do it, but the thing that put me off is
a) They are extremely bulky
b) They loose the property that makes them awesome, not be able to be stored in RAM.
Well now that the technology may allow it, I have also generated most of them (expect it to be about 50Gb without doing permutation and stuff). All this stuff is for the really enthusiastic EGTB people that don't mind downloading gigabytes of data. For me the 220mb five men already take care of the needs of engines superbly. Maybe someone can calculate the benefit of going from 5men RAM to 6men SSD, but is probably not much anyway. So there is nothing new here since 6 men EGBBs already existed.

Special cases like kBNK are taken care of just like engine's do it with a special king_square table. There are not so many cases like that so bitbases are enough 99.9% of the time, as evidenced by these results. So why waste extra space ? Also DTZ requires similar kind of help from the engines to avoid unnatural plays (thus requires effort from programmers). So is it really worth it to offer an 80Gb solution to a 0.1% problem of not being able to mate but just draw? That is a fair question.

[Laskos]

why I should invest an extra 80gb of space.

Compared to?

Existing bitbases solutions.

Syzygy WDL+DTZ 3-4-5 men take less than 1GB on my HD, so they are not much bulkier than RAM based EGBBs.

[Laskos]

Just tested Toga II v3.0 with Scorpio EGBBs (cache 128M), at 15+0.15 TC, LOS 99.9% stopping:

Code: Select all

    Program                              Score     %     Elo    +   -    Draws

  1 Toga Scorpio                   : 1931.0/3736  51.7     6    8   8   45.0 %
  2 Toga                           : 1805.0/3736  48.3    -6    8   8   45.0 %

12 +/- 8 Elo points benefit for Scorpio EGBBs. By now I tested Nalimov TBs (0+/-2), Shreddder EGBBs (14+/-8), Scorpio EGBBs (12+/-8) and Syzygy TBs (15+/-8). Within error margins, well implemented EGBBs give the same benefit as Syzygy TBs (at least to 3-4-5 men).

The main point for me of Syzygy bases is that they have 6 men WDL and that they are available now. All those tests where the Syzygy's are lobotomized by throwing out the 6 men don't mean too much except confirming that they don't suck. Besides, it is nice that they can be generated within a few days on a relatively simple computer, have a liberal license and unencumbered support. If you don't want the DTZ data, then just remove that and use heuristics to convert a win. (But I don't yet fully understand how pure WDL tables can convert KBNK into a mate. So for the time being I keep them. Disk space is cheap anyway).

I am in no mood to download 6 men anything. But I was surprised to see the benefit from 5-men bases, as previously, with Nalimovs, I had inconclusive tests ELO-wise.

[Daniel Shawul]

He is talking about 6 men:

Code: Select all

Shredder: 40Gb
Syzygy; 150Gb
Scorpio*: about 50Gb

So infact the difference is 110Gb extra space to shredder and 100Gb to scorpio. If we go by 5 men results, it looks like a total waste but one has to do the 6-men test to know since there are more cases that need attention. But Diep has used 6-men WDL alone so bitbases already has some support there too.

*In process of generation.

[mvk]

The main point of Syzygy bases is that they have 6 men WDL and that they are available now. All those tests where the Syzygy's are lobotomized by throwing out the 6 men don't mean too much except confirming that they don't suck. Besides, it is nice that they can be generated within a few days on a relatively simple computer, have a liberal license and unencumbered support. If you don't want the DTZ data, then just remove them and use heuristics to convert a win. (But I don't yet fully understand how pure WDL tables can convert KBNK into a mate. So for the time being I keep them. Disk space is cheap anyway).

Well shredder has 40GB bitbases! I could also have generated them 3 years ago since i had the code to do it, but the thing that put me off is
a) They are extremely bulky
b) They loose the property that makes them awesome, not be able to be stored in RAM.
Well now that the technology may allow it, I have also generated most of them (expect it to be about 50Gb without doing permutation and stuff). All this stuff is for the really enthusiastic EGTB people that don't mind downloading gigabytes of data. For me the 220mb five men already take care of the needs of engines superbly. Maybe someone can calculate the benefit of going from 5men RAM to 6men SSD, but is probably not much anyway. So there is nothing new here since 6 men EGBBs already existed.

It would be good to see your 6 men tables appear.

So for now the only alternative is then the Shredder bases? Unless I'm mistaken, I can't integrate those in my engine just the same? If they can, I wasn't aware of that and that is very nice of course.

I don't worry about the RAM issue for 6 men WDL btw. You typically wouldn't need all of it paged in at the same time.

Special cases like kBNK are taken care of just like engine's do it with a special king_square table. There are not so many cases like that so bitbases are enough 99.9% of the time, as evidenced by these results. So why waste extra space ? Also DTZ requires similar kind of help from the engines to avoid unnatural plays (thus requires effort from programmers). So is it really worth it to offer an 80Gb solution to a 0.1% problem of not being able to mate but just draw? That is a fair question.

For me the reason I prefer tables with pre-calculated solutions over heuristics is to avoid eg-specific code. That, and not having to worry about such end games to play a role in tuning. (The opposite question regarding 'waste' is why waste programmer time when you can lookup perfect information. Of course it is all recreational stuff, so the answer is a matter of taste)

[Daniel Shawul]

I am in no mood to download 6 men anything. But I was surprised to see the benefit from 5-men bases, as previously, with Nalimovs, I had inconclusive tests ELO-wise.

I totally understand that! Even i don't want to generate them but my hands were pretty much tied

[Daniel Shawul]

Ofcourse fseek()'ing here and there to a large number of files of Gb size is going to slow you down because of cache misses. The problem with Nalimov's 1.2TB relative to EGBBs is their size not allowing for efficient cacheing. So ofcouse RAM has a bigger role to play when you go from MB to 150GB. Most things you do to compress bitbses well them have a price later in probing, so you just have to make a trade off. For example, I decided to order my pieces in (PKNBRQ) order with the highest mobile piece placed last. The idea is that this will improve caching, but it has to be tested to be sure. Some like bitbases to be smaller instead, which is a good choice too.

[syzygy]

why I should invest an extra 80gb of space.

Compared to?

Existing bitbases solutions.

I provide a 68.2 GB 6-men "bitbase" solution. The 81.9 GB of DTZ is an optional addendum. I am not getting your point, I'm afraid.

Are you thinking of any other 6-men "bitbase" solution?

I only know of Robbobases. They are 100 GB or so with an optional 450 GB addendum.