6-men (64 bit) Nalimov EGTB generator

Discussion of anything and everything relating to chess playing software and machines.

Moderators: hgm, Harvey Williamson, bob

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
Gian-Carlo Pascutto
Posts: 1120
Joined: Sat Dec 13, 2008 6:00 pm
Contact:

6-men (64 bit) Nalimov EGTB generator

Post by Gian-Carlo Pascutto » Sun Sep 13, 2009 5:30 pm

Hi all,

I had a hard time finding Eugene's generator for 6 men, and when I found it, it was usually 32 bit only without source. I eventually found the sources for an incomplete Linux port on some obscure Russian site.

I've cleaned up this code so it now compiles on Windows again, and also completed the Linux port so it compiles with recent GCC's and works correctly in 64 bit mode.

Below are two optimized compiles (with source).

http://www.sjeng.org/ftp/tbgen_6m_x64.zip (Windows x64)
http://www.sjeng.org/ftp/tbgen_6m_lx64.tar.gz (Linux x64)

Some notes:
- This generator optionally uses a bitvector based algorithm. As far as I could tell, this streams in a bitmask with the relevant positions at the start of each iteration, does all the work, and then streams out the bitmask at the end of each iteration again. Main gain compared to the older algorithm is that there is no random disk seeking during the generation (=faster!), and that the memory needs are reduced by a factor 4. The algorithm is enabled by using the -b flag, with the size of a buffer in megabytes. From what I can see, you don't want to make this buffer too big, because that might reduce performance - the code will write out the entire buffer if it sees any bit in it changed. Something like 4 to 16 megabytes would be fine.
- It can now automatically determine the memory needed for generating an EGTB. Note that if it needs to generate dependent EGTB's, those might have higher requirements and fail with out of memory errors. (Example would be kpppk which has 3fold symmetry in the pawns, and hence a lower memory requirement than the dependant kqrpk.)
- The -p flag to map everything doesn't seem to be very useful for the new algorithm. Better to use -c with something like 256. The only time the cache is used is at the very start, in the "probing minor tables" step, and it's also the only time any random disk access is done.
- Aside from the streaming in and streaming out phase described at the start, the generator is fully parallel and can use all cores.

On my Q6600 with 6G RAM, I can generate a 6 man table in approximately a day. The limiting factor appears to be the disk streaming speed. Even at 70M/s read and/or write speed, it takes several minutes to read in or out the entire 10G tables. Iterating through all positions appears to be quite fast when using all 4 cores.

User avatar
AdminX
Posts: 5040
Joined: Mon Mar 13, 2006 1:34 pm
Location: Acworth, GA
Contact:

Re: 6-men (64 bit) Nalimov EGTB generator

Post by AdminX » Sun Sep 13, 2009 5:33 pm

Thanks much GCP! :D
"Good decisions come from experience, and experience comes from bad decisions."
__________________________________________________________________
Ted Summers

ernst
Posts: 328
Joined: Thu Mar 09, 2006 5:00 pm

Re: 6-men (64 bit) Nalimov EGTB generator

Post by ernst » Sun Sep 13, 2009 6:40 pm

Thanks Gian-Carlo. The output is uncompressed or compressed?

Gian-Carlo Pascutto
Posts: 1120
Joined: Sat Dec 13, 2008 6:00 pm
Contact:

Re: 6-men (64 bit) Nalimov EGTB generator

Post by Gian-Carlo Pascutto » Sun Sep 13, 2009 6:41 pm

Uncompressed, but they can be compressed with the datacomp program.

frankp
Posts: 216
Joined: Sun Mar 12, 2006 2:11 pm

Re: 6-men (64 bit) Nalimov EGTB generator

Post by frankp » Mon Sep 14, 2009 7:10 pm

Do you need the five man uncompressed files?

Gian-Carlo Pascutto
Posts: 1120
Joined: Sat Dec 13, 2008 6:00 pm
Contact:

Re: 6-men (64 bit) Nalimov EGTB generator

Post by Gian-Carlo Pascutto » Mon Sep 14, 2009 10:25 pm

frankp wrote:Do you need the five man uncompressed files?
No, anything but the table currently being built can be left compressed.

User avatar
Sylwy
Posts: 3358
Joined: Fri Apr 21, 2006 2:19 pm
Location: IASI (Romania) - the historical capital of MOLDOVA

Re: Obscure site ???

Post by Sylwy » Tue Sep 15, 2009 6:09 am

Gian-Carlo Pascutto wrote:.........on some obscure Russian site.

:lol: :lol: :lol: :lol: :lol: :lol: :lol: ......... :lol: :lol: :lol: :lol: :lol: :lol:

Gian-Carlo Pascutto
Posts: 1120
Joined: Sat Dec 13, 2008 6:00 pm
Contact:

Re: 6-men (64 bit) Nalimov EGTB generator

Post by Gian-Carlo Pascutto » Tue Sep 15, 2009 9:21 pm

Hmm, some EGTB are failing verification here.

I'll have to see whether it's a 64 bit issue, a bug or if something was miscompiled.

frankp
Posts: 216
Joined: Sun Mar 12, 2006 2:11 pm

Re: 6-men (64 bit) Nalimov EGTB generator

Post by frankp » Wed Sep 16, 2009 4:12 pm

I generated the five man egtbs and compressed them. kppkp failed. Everything else passed. (Repeated kppkp generated and compressed, with same result.).

vb4
Posts: 149
Joined: Sat Mar 11, 2006 4:45 am
Location: NY

Re: 6-men (64 bit) Nalimov EGTB generator

Post by vb4 » Thu Sep 17, 2009 4:05 am

Hi GCP,

Great work!! Question, will this generator generate the stat files that were customary with Eugenes generator?

Thanks

Post Reply