New 6-piece tablebases

Discussion of chess software programming and technical issues.

Moderators: hgm, Harvey Williamson, bob

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
syzygy
Posts: 4343
Joined: Tue Feb 28, 2012 10:56 pm

Re: New 6-piece tablebases

Post by syzygy » Mon Apr 08, 2013 8:58 pm

Sven Schüle wrote:
jshriver wrote:I can't speak for anyone else, but personally I would rather have a 7men that is horrible slow, disk crunching, but works, than another 5-10 years of none :)
If generation of all 6-men tables in RAM took "less than 5 days" with Ronald's generator (so probably more than 4 days) then generation of all 7-men tables in RAM (if the required hardware were available) would take several years, you know why.
I think it took close to 5 days. The complete 3-, 4- and 5-men sets took about 75 minutes. This includes the time spent on compression, which is relatively more efficient for larger tables (since the part of it that isn't multithreaded is essentially independent of table size).
Now what should we expect from the much slower, disk-based approach? Maybe you would see the first 7-men table done after those 5-10 years ...
Since 7-men tables have been generated on relatively modest hardware, it is probably a bit better than that, and a distributed approach would be possible, but then you have the problem of distributing the resulting tables. In my view it's just a different game. Since I wouldn't be the first to generate them by a long shot, I'm not going to give it a try until it is practically feasible to do a complete set by myself. (Well, you never know... maybe one day I'll be inspired to write a disk-based generator just for the heck of it.)

User avatar
jshriver
Posts: 1105
Joined: Wed Mar 08, 2006 8:41 pm
Location: Morgantown, WV, USA
Contact:

Re: New 6-piece tablebases

Post by jshriver » Tue Apr 09, 2013 4:45 am

Do you have a md5sum of the entire 3-4-5 set?

Tried generating and verifying twice and having a couple hiccups. Looks like the final set is around 1.1gigs though.

-Josh

Edit: Think the problem I'm having is run.pl during verify is looking for x.rtbz but instead when they are being generating I'm getting x.rtbz.N where n is 1..4

syzygy
Posts: 4343
Joined: Tue Feb 28, 2012 10:56 pm

Re: New 6-piece tablebases

Post by syzygy » Tue Apr 09, 2013 7:40 am

jshriver wrote:Edit: Think the problem I'm having is run.pl during verify is looking for x.rtbz but instead when they are being generating I'm getting x.rtbz.N where n is 1..4
Then the generator has crashed for some reason (or was interrupted). The .N files are the results per file which are in the end merged together. If all goes fine you just end up with .rtbw and .rtbz.

The final set should be below 1.1GB, but I am counting 1 GB = 1024 x 1024 x 1024.

syzygy
Posts: 4343
Joined: Tue Feb 28, 2012 10:56 pm

Re: New 6-piece tablebases

Post by syzygy » Tue Apr 09, 2013 8:04 am

Did you at the end have the files for KPPPvK and KPPvKP?

If I have time today I'll regenerate the 3-4-5-set myself to see if I have any problems with the latest code. (I don't have any for 3-4.)

Alexander Zacharias
Posts: 5
Joined: Thu May 31, 2012 3:59 pm
Location: Germany

Re: New 6-piece tablebases

Post by Alexander Zacharias » Tue Apr 09, 2013 6:10 pm

I'm quite enthusiastic about this, though I don't have the necessary RAM at hand yet.
Do you have performance data? What playing strength did your program gain by using the 6-man tables? Can you estimate the gain for Stockfish?

ZirconiumX
Posts: 1327
Joined: Sun Jul 17, 2011 9:14 am

Re: New 6-piece tablebases

Post by ZirconiumX » Tue Apr 09, 2013 8:07 pm

Alexander Zacharias wrote:I'm quite enthusiastic about this, though I don't have the necessary RAM at hand yet.
Do you have performance data? What playing strength did your program gain by using the 6-man tables? Can you estimate the gain for Stockfish?
What do you mean by performance data??? The code is very quick, and works well.

Don't expect a stockfish version any time soon.

Ron - what are these magic tablebases called?

Matthew:out
Some believe in the almighty dollar.

I believe in the almighty printf statement.

Alexander Zacharias
Posts: 5
Joined: Thu May 31, 2012 3:59 pm
Location: Germany

Re: New 6-piece tablebases

Post by Alexander Zacharias » Tue Apr 09, 2013 10:41 pm

ZirconiumX wrote: What do you mean by performance data??? The code is very quick, and works well.
I originally thought of performance as game-playing-performance, measured in ELO. Though it would be interesting to know how fast the access is if the needed part of the tablebase is already in RAM.
ZirconiumX wrote: Don't expect a stockfish version any time soon.
Ronalds post and a quick look at his github repository gave me the impression that he was already able to let Stockfish play full games with his tablebase access. Did I get something wrong?
Ronald de Man wrote: As a proof of concept I have adapted the probing code to Stockfish.
Greetings,
Alexander

syzygy
Posts: 4343
Joined: Tue Feb 28, 2012 10:56 pm

Re: New 6-piece tablebases

Post by syzygy » Tue Apr 09, 2013 10:44 pm

Alexander Zacharias wrote:I'm quite enthusiastic about this, though I don't have the necessary RAM at hand yet.
Do you have performance data? What playing strength did your program gain by using the 6-man tables? Can you estimate the gain for Stockfish?
I've not tried to measure the elo increase for my engine, but my unscientific impression from watching it play against engines of comparable strength on FICS is that it saves quite a few half points. Performance will depend on available system RAM for caching and speed of HD / SSD. Since I have them on SSD I'm not seeing significant nps drops anymore.

syzygy
Posts: 4343
Joined: Tue Feb 28, 2012 10:56 pm

Re: New 6-piece tablebases

Post by syzygy » Wed Apr 10, 2013 12:29 am

jshriver wrote:Tried generating and verifying twice and having a couple hiccups.
I've regenerated the 3-4-5 set without any problem. However, I did find a bug that crashed tbcheck, which might also explain crashes of rtbgen and rtbgenp. It's fixed now. It should not have let to any verification error.

Total number of bytes is 396105616 (wc -c *.rtbw) and 587852880 (wc -c *.rtbz).

md5sums:

Code: Select all

$ md5sum *.rtbw | md5sum
79d415421114a4861b8f80cf0064863f  -
$ md5sum *.rtbz | md5sum
095c18a7e8df1a629c8518e1509fccad  -

syzygy
Posts: 4343
Joined: Tue Feb 28, 2012 10:56 pm

Re: New 6-piece tablebases

Post by syzygy » Wed Apr 10, 2013 12:35 am

Alexander Zacharias wrote:
ZirconiumX wrote: Don't expect a stockfish version any time soon.
Ronalds post and a quick look at his github repository gave me the impression that he was already able to let Stockfish play full games with his tablebase access. Did I get something wrong?
I've included some files that can be copied over Stockfish source from github to add probing code to it (after adding tbprobe.o to OBJS in the Makefile). It should now also work on Windows. My testing of this has been very limited though; it's only intended as an example.
ZirconiumX wrote:Ron - what are these magic tablebases called?
Good question. I've always called my engine Sjaak, but this name is not unique anymore. It's playing on FICS as TrojanKnight. TrojanKnight tablebases? Trojanbases?

Post Reply