New 6-piece tablebases

[syzygy]

I can't speak for anyone else, but personally I would rather have a 7men that is horrible slow, disk crunching, but works, than another 5-10 years of none

If generation of all 6-men tables in RAM took "less than 5 days" with Ronald's generator (so probably more than 4 days) then generation of all 7-men tables in RAM (if the required hardware were available) would take several years, you know why.

I think it took close to 5 days. The complete 3-, 4- and 5-men sets took about 75 minutes. This includes the time spent on compression, which is relatively more efficient for larger tables (since the part of it that isn't multithreaded is essentially independent of table size).

Now what should we expect from the much slower, disk-based approach? Maybe you would see the first 7-men table done after those 5-10 years ...

Since 7-men tables have been generated on relatively modest hardware, it is probably a bit better than that, and a distributed approach would be possible, but then you have the problem of distributing the resulting tables. In my view it's just a different game. Since I wouldn't be the first to generate them by a long shot, I'm not going to give it a try until it is practically feasible to do a complete set by myself. (Well, you never know... maybe one day I'll be inspired to write a disk-based generator just for the heck of it.)

[jshriver]

Do you have a md5sum of the entire 3-4-5 set?

Tried generating and verifying twice and having a couple hiccups. Looks like the final set is around 1.1gigs though.

-Josh

Edit: Think the problem I'm having is run.pl during verify is looking for x.rtbz but instead when they are being generating I'm getting x.rtbz.N where n is 1..4

[syzygy]

Edit: Think the problem I'm having is run.pl during verify is looking for x.rtbz but instead when they are being generating I'm getting x.rtbz.N where n is 1..4

Then the generator has crashed for some reason (or was interrupted). The .N files are the results per file which are in the end merged together. If all goes fine you just end up with .rtbw and .rtbz.

The final set should be below 1.1GB, but I am counting 1 GB = 1024 x 1024 x 1024.

[syzygy]

Did you at the end have the files for KPPPvK and KPPvKP?

If I have time today I'll regenerate the 3-4-5-set myself to see if I have any problems with the latest code. (I don't have any for 3-4.)

[Alexander Zacharias]

I'm quite enthusiastic about this, though I don't have the necessary RAM at hand yet.
Do you have performance data? What playing strength did your program gain by using the 6-man tables? Can you estimate the gain for Stockfish?

[ZirconiumX]

I'm quite enthusiastic about this, though I don't have the necessary RAM at hand yet.
Do you have performance data? What playing strength did your program gain by using the 6-man tables? Can you estimate the gain for Stockfish?

What do you mean by performance data??? The code is very quick, and works well.

Don't expect a stockfish version any time soon.

Ron - what are these magic tablebases called?

Matthew:out

[Alexander Zacharias]

What do you mean by performance data??? The code is very quick, and works well.

I originally thought of performance as game-playing-performance, measured in ELO. Though it would be interesting to know how fast the access is if the needed part of the tablebase is already in RAM.

Don't expect a stockfish version any time soon.

Ronalds post and a quick look at his github repository gave me the impression that he was already able to let Stockfish play full games with his tablebase access. Did I get something wrong?

As a proof of concept I have adapted the probing code to Stockfish.

Greetings,
Alexander

[syzygy]

I'm quite enthusiastic about this, though I don't have the necessary RAM at hand yet.
Do you have performance data? What playing strength did your program gain by using the 6-man tables? Can you estimate the gain for Stockfish?

I've not tried to measure the elo increase for my engine, but my unscientific impression from watching it play against engines of comparable strength on FICS is that it saves quite a few half points. Performance will depend on available system RAM for caching and speed of HD / SSD. Since I have them on SSD I'm not seeing significant nps drops anymore.

[syzygy]

Tried generating and verifying twice and having a couple hiccups.

I've regenerated the 3-4-5 set without any problem. However, I did find a bug that crashed tbcheck, which might also explain crashes of rtbgen and rtbgenp. It's fixed now. It should not have let to any verification error.

Total number of bytes is 396105616 (wc -c *.rtbw) and 587852880 (wc -c *.rtbz).

md5sums:

Code: Select all

$ md5sum *.rtbw | md5sum
79d415421114a4861b8f80cf0064863f  -
$ md5sum *.rtbz | md5sum
095c18a7e8df1a629c8518e1509fccad  -

[syzygy]

Don't expect a stockfish version any time soon.

Ronalds post and a quick look at his github repository gave me the impression that he was already able to let Stockfish play full games with his tablebase access. Did I get something wrong?

I've included some files that can be copied over Stockfish source from github to add probing code to it (after adding tbprobe.o to OBJS in the Makefile). It should now also work on Windows. My testing of this has been very limited though; it's only intended as an example.

Ron - what are these magic tablebases called?

Good question. I've always called my engine Sjaak, but this name is not unique anymore. It's playing on FICS as TrojanKnight. TrojanKnight tablebases? Trojanbases?