New 6-piece tablebases

Discussion of chess software programming and technical issues.

Moderators: hgm, Harvey Williamson, bob

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
User avatar
Houdini
Posts: 1471
Joined: Mon Mar 15, 2010 11:00 pm
Contact:

Re: New 6-piece tablebases

Post by Houdini » Sat Apr 06, 2013 11:04 pm

Daniel Shawul wrote:I decide to keep them in based on what i have. Stop spreading FUD or own up to your claims.
You have an interesting way of dealing with user feedback.

syzygy
Posts: 4343
Joined: Tue Feb 28, 2012 10:56 pm

Re: New 6-piece tablebases

Post by syzygy » Sun Apr 07, 2013 11:19 pm

The generator now also compiles on 64-bit Windows using MinGW.
It seems to work correctly at least up to 5 pieces. My Windows laptop does not have sufficient memory to test the generation of 6-piece tables. On Linux using wine it fails to allocate the required memory, but that could be the fault of wine.

User avatar
jshriver
Posts: 1105
Joined: Wed Mar 08, 2006 8:41 pm
Location: Morgantown, WV, USA
Contact:

Re: New 6-piece tablebases

Post by jshriver » Mon Apr 08, 2013 2:51 pm

Brilliant. I'm working on the gen now! Thanks for your hard work and contribution to the community.

-Josh

P.S. Any thoughts on expanding to 7-men?

syzygy
Posts: 4343
Joined: Tue Feb 28, 2012 10:56 pm

Re: New 6-piece tablebases

Post by syzygy » Mon Apr 08, 2013 7:17 pm

jshriver wrote:I'm working on the gen now!
Good! On Linux or on Windows? How much RAM does your system have?
P.S. Any thoughts on expanding to 7-men?
With minor modifications it could generate 7-men tables, but the machine to run it would need about 1628 GB of RAM fully addressable by all processors for pawnless tables ;-). Less for tables with pawns, since those are generated file by file. With a small change the requirement for pawnless could be lowered to about 1280 GB.

So in reality going to 7-men either means a disk-based approach or a cluster-based approach with quite big hardware. Disk-based will be slow and will probably require a complete redesign of the generation and compression algorithms. Cluster-based will also require significant changes, but might fit the current algorithms better.

Anyway, this is not in my plans. What I might do is extend the generator to do 7-piece tables for positions with two opposing (but not necessarily blocked) pawns. Those don't require the 7-piece pawnless tables and can be done file by file or split up even further.

User avatar
jshriver
Posts: 1105
Joined: Wed Mar 08, 2006 8:41 pm
Location: Morgantown, WV, USA
Contact:

Re: New 6-piece tablebases

Post by jshriver » Mon Apr 08, 2013 7:22 pm

I can't speak for anyone else, but personally I would rather have a 7men that is horrible slow, disk crunching, but works, than another 5-10 years of none :)

When I started my interest 5-6+ years ago, it seemed 7men was possible but nothing really came about because people were wanting something super efficient. Sadly even if it wasn't efficient, if we *had* something back then it might be completed now.

Just my $0.02 :)

Greatly appreciate your work and looking forward to checking it out further.

-Josh

User avatar
jshriver
Posts: 1105
Joined: Wed Mar 08, 2006 8:41 pm
Location: Morgantown, WV, USA
Contact:

Re: New 6-piece tablebases

Post by jshriver » Mon Apr 08, 2013 7:23 pm

syzygy wrote: Good! On Linux or on Windows? How much RAM does your system have?
16gigs DDR3 Ubuntu Linux.

Though this might be the push I need to get another couple 8gig sticks so I can generate the 6men set.

If you don't mind, I'd like to host the 3-4-5 dataset on my site like I do the Gaviota set.

-Josh

syzygy
Posts: 4343
Joined: Tue Feb 28, 2012 10:56 pm

Re: New 6-piece tablebases

Post by syzygy » Mon Apr 08, 2013 8:19 pm

jshriver wrote:I can't speak for anyone else, but personally I would rather have a 7men that is horrible slow, disk crunching, but works, than another 5-10 years of none :)
Sergey Yankovich wrote such a generator. From what I have read it is able to generate 7-men tables on commodity hardware using months of heavy disk access per table. (There are some posts on this by "De Knaller" on the Rybka forum.)
When I started my interest 5-6+ years ago, it seemed 7men was possible but nothing really came about because people were wanting something super efficient. Sadly even if it wasn't efficient, if we *had* something back then it might be completed now.
It is possible to generate single tables, but a complete 7-men set is still too big to be stored, distributed or accessed during search with any efficiency. With massive resources it is of course possible, as was shown by the people behind the Lomonosov tables.

Don't get me wrong, I have nothing against the generation of 7-, 8- or 9-men tables, but my goal was to have a complete set that is practically usable. (In fact I want a complete set of 6-men tables for suicide chess, which means 5754 6-men tables in total. So generation has to be fast and compression has to be good.)

If hardware keeps progressing, with time 7-men will become practically usable and my code will only need a recompilation.

syzygy
Posts: 4343
Joined: Tue Feb 28, 2012 10:56 pm

Re: New 6-piece tablebases

Post by syzygy » Mon Apr 08, 2013 8:27 pm

jshriver wrote:
syzygy wrote: Good! On Linux or on Windows? How much RAM does your system have?
16gigs DDR3 Ubuntu Linux.

Though this might be the push I need to get another couple 8gig sticks so I can generate the 6men set.
For the pawnless 6-men tables that seems a good idea. 16 GB might about suffice for the tables with pawns, but of course you can't generate those without the pawnless tables. I think tables without pawns need about 26-27 GB. I might look into adding an option that lowers this to 20-21 GB so that 24 GB is enough.
If you don't mind, I'd like to host the 3-4-5 dataset on my site like I do the Gaviota set.
I certainly don't mind!

Sven
Posts: 3704
Joined: Thu May 15, 2008 7:57 pm
Location: Berlin, Germany
Full name: Sven Schüle
Contact:

Re: New 6-piece tablebases

Post by Sven » Mon Apr 08, 2013 8:34 pm

jshriver wrote:I can't speak for anyone else, but personally I would rather have a 7men that is horrible slow, disk crunching, but works, than another 5-10 years of none :)
If generation of all 6-men tables in RAM took "less than 5 days" with Ronald's generator (so probably more than 4 days) then generation of all 7-men tables in RAM (if the required hardware were available) would take several years, you know why.

Now what should we expect from the much slower, disk-based approach? Maybe you would see the first 7-men table done after those 5-10 years ...

Sven

User avatar
jshriver
Posts: 1105
Joined: Wed Mar 08, 2006 8:41 pm
Location: Morgantown, WV, USA
Contact:

Re: New 6-piece tablebases

Post by jshriver » Mon Apr 08, 2013 8:42 pm

Sven Schüle wrote:
Now what should we expect from the much slower, disk-based approach? Maybe you would see the first 7-men table done after those 5-10 years ...

Sven
Aye, understand the logic behind it. Just been itching for 7-men data for nearly a decade now. Meant no disrespect toward Mr Ronald de Man. Was just curious :).

Post Reply