New 6-piece tablebases

Discussion of chess software programming and technical issues.

Moderators: hgm, Rebel, chrisw

phenri
Posts: 284
Joined: Tue Aug 13, 2013 9:44 am

Re: New 6-piece tablebases

Post by phenri »

jshriver wrote:I finished generating the full 3-4-5-6 men set in both WDL and DTZ. With an i7-3770 with 32gigs RAM it took 14 days, followed by just under 4 days to do a verify.

(...)
To download the complete 3-4-5 as a single file use this:
(...)
Hello Josh,
thanks for your time.

I made a mirror of sbases345.tar.bz2

File http://ldd6qz1x.1fichier.com/en/ sbases345.tar.bz2
Size 949616772 B
Checksum (Whirlpool)

Code: Select all

5672e68a810e143f40ea06dfd8cd0feef4237943150073fe807198509075d70a2d7509277c0c467ea579e6f2339b23aced1f9e078bae44d33ede1ddb729789c7
User avatar
Kirill Kryukov
Posts: 492
Joined: Sun Mar 19, 2006 4:12 am
Full name: Kirill Kryukov

Re: New 6-piece tablebases

Post by Kirill Kryukov »

syzygy wrote:
Kirill Kryukov wrote:I finally had the time to try your generator. I could build the 3-4-5 piece tablebases without problems, and all checksums matched yours. However "rtbgen" crashes every time I try to build any 6-piece tablebase - both standalone and via "run.pl". It crashes instantly with no useful diagnostics.

Code: Select all

      0 [main] rtbgen 7224 open_stackdumpfile: Dumping stack trace to rtbgen.exe.stackdump
Did you try the very latest code? It has worked for others, at least with (MingW) gcc-4.7.2.
Does rtbgen.exe.stackdump contain anything?
Another point, GCC printed a bunch or warnings. I humbly suggest that it's perhaps good idea to try and eliminate all warnings, so that the code will give better impression and inspire more confidence.
All array bounds warnings are a bug of gcc-4.8.x. The solution is to add -Wno-array-bounds. The "may be used uninitialized" warnings are all bogus.

Code: Select all

tbgen.c:522:14: warning: ‘optarg’ redeclared without dllimport attribute: previous dllimport ignored [-Wattributes]
 extern char *optarg;
I never saw this warning. I suppose the line could/should be removed.
I will try to build it with MinGW around weekend and see if it works better.
Ah! I never tested with cygwin. It probably can't deal with very large memory allocations.
I recompiled your generator with MinGW GCC 4.8.1 (actually it is also from cygwin64 distribution), and it seems working so far. I generated KQQQQvK successfully. After my other computation completes I'm going to proceed generating all 6-piece tablebases, I'll then report on my progress or any issues.

Ronald, are you interested in setting up any centralized distribution for these tablebases? E.g., a torrent, for those who can't (or don't want to) generate them? I could contribute bandwidth or perhaps host the tracker. (Actually anyone other than Ronald is also welcome to comment / participate).
User avatar
jshriver
Posts: 1354
Joined: Wed Mar 08, 2006 9:41 pm
Location: Morgantown, WV, USA

Re: New 6-piece tablebases

Post by jshriver »

Kirill Kryukov wrote: Ronald, are you interested in setting up any centralized distribution for these tablebases? E.g., a torrent, for those who can't (or don't want to) generate them? I could contribute bandwidth or perhaps host the tracker. (Actually anyone other than Ronald is also welcome to comment / participate).
I'm going to see if I can host them via http. If not I was going to lean toward running my own torrent tracker, and then seeding them from 1-2 sources.

Due to work and some family health concerns I'm a bit tied up till this weekend.
User avatar
Kirill Kryukov
Posts: 492
Joined: Sun Mar 19, 2006 4:12 am
Full name: Kirill Kryukov

Re: New 6-piece tablebases

Post by Kirill Kryukov »

jshriver wrote:
Kirill Kryukov wrote: Ronald, are you interested in setting up any centralized distribution for these tablebases? E.g., a torrent, for those who can't (or don't want to) generate them? I could contribute bandwidth or perhaps host the tracker. (Actually anyone other than Ronald is also welcome to comment / participate).
I'm going to see if I can host them via http. If not I was going to lean toward running my own torrent tracker, and then seeding them from 1-2 sources.

Due to work and some family health concerns I'm a bit tied up till this weekend.
Josh, that would be very nice. I prefer a torrent rather than http so that others (like myself) can contribute. 151 GB is a lot to host by yourself.
User avatar
jshriver
Posts: 1354
Joined: Wed Mar 08, 2006 9:41 pm
Location: Morgantown, WV, USA

Re: New 6-piece tablebases

Post by jshriver »

Kirill Kryukov wrote: Josh, that would be very nice. I prefer a torrent rather than http so that others (like myself) can contribute. 151 GB is a lot to host by yourself.
http://oics.olympuschess.com/tracker/index.php

This is my first time really doing this. But willing to give it a try. Creating a torrent file now of the sbases345.tar.gz
User avatar
Kirill Kryukov
Posts: 492
Joined: Sun Mar 19, 2006 4:12 am
Full name: Kirill Kryukov

Re: New 6-piece tablebases

Post by Kirill Kryukov »

jshriver wrote:
Kirill Kryukov wrote: Josh, that would be very nice. I prefer a torrent rather than http so that others (like myself) can contribute. 151 GB is a lot to host by yourself.
http://oics.olympuschess.com/tracker/index.php

This is my first time really doing this. But willing to give it a try. Creating a torrent file now of the sbases345.tar.gz
Looks good, but you should share the actual tablebase files instead of the tar.bz2 archive. This way people may keep sharing after downloading them. Any huge archive will be certainly deleted by users after downloading, defeating the purpose of torrent-sharing.
jpqy
Posts: 551
Joined: Thu Apr 24, 2008 9:31 am
Location: Belgium

Re: New 6-piece tablebases

Post by jpqy »

Nice and clean done Joshua :!:

Hope to see 6 pieces also like that guys..so that we can start download them!

Kind regards,
JP.
User avatar
jshriver
Posts: 1354
Joined: Wed Mar 08, 2006 9:41 pm
Location: Morgantown, WV, USA

Re: New 6-piece tablebases

Post by jshriver »

jpqy wrote:Nice and clean done Joshua :!:

Hope to see 6 pieces also like that guys..so that we can start download them!

Kind regards,
JP.
Thanks, having to learn as I go. Looks like seeding the big tarball is working, but the folder with multiple files is acting quirky. So will try and get that working. Once that show's successful, it should be easy to create one for the huge 6 piece set.

BTW would someone try downloading that sbases from my torrent? Curious how well this works on a smaller file. This is the same server I run OICS so I don't want to stress it to much.

Curious about the webseeding though. When I created the torrent for all the 3-4-5 "individual files" I also gave it the address to where I'm hosting them via the web, though it doesn't seem to be pulling them in.
IQ
Posts: 162
Joined: Thu Dec 17, 2009 10:46 am

Re: New 6-piece tablebases

Post by IQ »

jshriver wrote:
jpqy wrote:Nice and clean done Joshua :!:

Hope to see 6 pieces also like that guys..so that we can start download them!

Kind regards,
JP.
Thanks, having to learn as I go. Looks like seeding the big tarball is working, but the folder with multiple files is acting quirky. So will try and get that working. Once that show's successful, it should be easy to create one for the huge 6 piece set.

BTW would someone try downloading that sbases from my torrent? Curious how well this works on a smaller file. This is the same server I run OICS so I don't want to stress it to much.

Curious about the webseeding though. When I created the torrent for all the 3-4-5 "individual files" I also gave it the address to where I'm hosting them via the web, though it doesn't seem to be pulling them in.
I am downloading both sbases345.tar.bz2 and syzygy right now from your torrent page. Seems to work pretty good - can't wait for the 6 pieces.
Vinvin
Posts: 5236
Joined: Thu Mar 09, 2006 9:40 am
Full name: Vincent Lejeune

Re: New 6-piece tablebases

Post by Vinvin »

phenri wrote: I made a mirror of sbases345.tar.bz2

Size 949616772 B
Nalimov 3-4-5 pieces is around 7 GB, does that means this new format is 1/7 the size of Nalimov TB ? :shock: