6-men (64 bit) Nalimov EGTB generator

[Gian-Carlo Pascutto]

Hi all,

I had a hard time finding Eugene's generator for 6 men, and when I found it, it was usually 32 bit only without source. I eventually found the sources for an incomplete Linux port on some obscure Russian site.

I've cleaned up this code so it now compiles on Windows again, and also completed the Linux port so it compiles with recent GCC's and works correctly in 64 bit mode.

Below are two optimized compiles (with source).

http://www.sjeng.org/ftp/tbgen_6m_x64.zip (Windows x64)
http://www.sjeng.org/ftp/tbgen_6m_lx64.tar.gz (Linux x64)

Some notes:
- This generator optionally uses a bitvector based algorithm. As far as I could tell, this streams in a bitmask with the relevant positions at the start of each iteration, does all the work, and then streams out the bitmask at the end of each iteration again. Main gain compared to the older algorithm is that there is no random disk seeking during the generation (=faster!), and that the memory needs are reduced by a factor 4. The algorithm is enabled by using the -b flag, with the size of a buffer in megabytes. From what I can see, you don't want to make this buffer too big, because that might reduce performance - the code will write out the entire buffer if it sees any bit in it changed. Something like 4 to 16 megabytes would be fine.
- It can now automatically determine the memory needed for generating an EGTB. Note that if it needs to generate dependent EGTB's, those might have higher requirements and fail with out of memory errors. (Example would be kpppk which has 3fold symmetry in the pawns, and hence a lower memory requirement than the dependant kqrpk.)
- The -p flag to map everything doesn't seem to be very useful for the new algorithm. Better to use -c with something like 256. The only time the cache is used is at the very start, in the "probing minor tables" step, and it's also the only time any random disk access is done.
- Aside from the streaming in and streaming out phase described at the start, the generator is fully parallel and can use all cores.

On my Q6600 with 6G RAM, I can generate a 6 man table in approximately a day. The limiting factor appears to be the disk streaming speed. Even at 70M/s read and/or write speed, it takes several minutes to read in or out the entire 10G tables. Iterating through all positions appears to be quite fast when using all 4 cores.

[AdminX]

Thanks much GCP!

[ernst]

Thanks Gian-Carlo. The output is uncompressed or compressed?

[Gian-Carlo Pascutto]

Uncompressed, but they can be compressed with the datacomp program.

[frankp]

Do you need the five man uncompressed files?

[Gian-Carlo Pascutto]

Do you need the five man uncompressed files?

No, anything but the table currently being built can be left compressed.

[Sylwy]

.........on some obscure Russian site.

.........

[Gian-Carlo Pascutto]

Hmm, some EGTB are failing verification here.

I'll have to see whether it's a 64 bit issue, a bug or if something was miscompiled.

[frankp]

I generated the five man egtbs and compressed them. kppkp failed. Everything else passed. (Repeated kppkp generated and compressed, with same result.).

[vb4]

Hi GCP,

Great work!! Question, will this generator generate the stat files that were customary with Eugenes generator?

Thanks