New 6-piece tablebases

[syzygy]

Looking forward to a bug free, DTZ working version

The info quoted by Kirill has all the information you need, and I already gave you the information several times. I am not a customer service desk, and this is the "Programming and Technical Discussions" part of the forum. Thank you.

[Sharaf_DG]

My apologies for any misunderstanding...Keep Up the Good work!!!

[gotogo]

why is it call new 6 piece tablebases when only 5 piece set is available?
anyone want to buy a police car?

[syzygy]

why is it call new 6 piece tablebases when only 5 piece set is available?

I am not even offering the 5 piece set for download.

I am only providing the source code of a tablebase generator that is capable of generating the 6 piece set. This is only intended for experienced users who are comfortable with command line interfaces and compiling their own binaries.

Anyone who wishes can try to generate the full 6 piece set and make it available for download. It is 150 GB, much smaller than the 1.2 TB of Nalimov but still big.

Another idea is developing a GUI frontend for the generator that would allow end users to generate the full set with a single click. On a fast 4-core cpu with 16 GB of RAM it should take less than 10 days. This might be more convenient than a 150 GB download.

[hgm]

Another idea is developing a GUI frontend for the generator that would allow end users to generate the full set with a single click. On a fast 4-core cpu with 16 GB of RAM it should take less than 10 days. This might be more convenient than a 150 GB download.

If you want a single-click interface, a simple .BAT file containing the required commands would do.

[syzygy]

Another idea is developing a GUI frontend for the generator that would allow end users to generate the full set with a single click. On a fast 4-core cpu with 16 GB of RAM it should take less than 10 days. This might be more convenient than a 150 GB download.

If you want a single-click interface, a simple .BAT file containing the required commands would do.

It is a bit more complicated, since a few parameters need to be set (number of threads, whether the system has 'only' 16 GB or at least 24 GB, whether to generate the 6-piece set or just the 5-piece set).

Btw, do you happen to know how to test (in a .BAT file) for the existence of a file and execute a command only if the file does not exist? This is necessary for being able to continue an interrupted generation process from where it left off. Currently I use a Perl script, but on Windows that requires installation of Perl.

[Kirill Kryukov]

Btw, do you happen to know how to test (in a .BAT file) for the existence of a file and execute a command only if the file does not exist? This is necessary for being able to continue an interrupted generation process from where it left off. Currently I use a Perl script, but on Windows that requires installation of Perl.

Code: Select all

if not exist <file> <command>

or type "if /?" for more details.

[Kirill Kryukov]

Hi Ronald,

I finally had the time to try your generator. I could build the 3-4-5 piece tablebases without problems, and all checksums matched yours. However "rtbgen" crashes every time I try to build any 6-piece tablebase - both standalone and via "run.pl". It crashes instantly with no useful diagnostics.

Code: Select all

      0 [main] rtbgen 7224 open_stackdumpfile: Dumping stack trace to rtbgen.exe.stackdump

So far I only tried it with GCC 4.8.1 (cygwin64) on Windows 7. The machine has 16 GB of RAM, the "--disk" option was always supplied to both generators and to "run.pl".

Another point, GCC printed a bunch or warnings. I humbly suggest that it's perhaps good idea to try and eliminate all warnings, so that the code will give better impression and inspire more confidence.

Code: Select all

tbgen.c:522:14: warning: ‘optarg’ redeclared without dllimport attribute: previous dllimport ignored [-Wattributes]
 extern char *optarg;
              ^
In file included from tbgen.c:17:0:
probe.c: In function ‘encode_piece’:
probe.c:821:7: warning: ‘idx’ may be used uninitialized in this function [-Wmaybe-uninitialized]
   idx *= factor[0];
       ^
probe.c: In function ‘decode_pawn’:
probe.c:1640:20: warning: array subscript is above array bounds [-Warray-bounds]
  if (sort[j] > sort[k]) Swap(sort[j], sort[k]);
                    ^
probe.c:1640:43: warning: array subscript is above array bounds [-Warray-bounds]
  if (sort[j] > sort[k]) Swap(sort[j], sort[k]);
                                           ^
permute.c: In function ‘init_permute_piece’:
permute.c:919:26: warning: array subscript is above array bounds [-Warray-bounds]
        piece_perm_list[i][m++] = tidx[type_perm_list[i][k]] + l;
                          ^
permute.c:919:26: warning: array subscript is above array bounds [-Warray-bounds]
permute.c:919:26: warning: array subscript is above array bounds [-Warray-bounds]
permute.c:942:26: warning: array subscript is above array bounds [-Warray-bounds]
        piece_perm_list[i][m++] = tidx[type_perm_list[i][k]] + l;
                          ^
permute.c:942:26: warning: array subscript is above array bounds [-Warray-bounds]
compress.c: In function ‘remove_wdl_worker’:
compress.c:473:10: warning: ‘t’ may be used uninitialized in this function [-Wmaybe-uninitialized]
   int s, t;
          ^
compress.c: In function ‘adjust_work_replace’:
compress.c:427:15: warning: ‘idx2’ may be used uninitialized in this function [-Wmaybe-uninitialized]
       work[i] = idx2;
               ^
tbgenp.c:627:14: warning: ‘optarg’ redeclared without dllimport attribute: previous dllimport ignored [-Wattributes]
 extern char *optarg;
              ^
In file included from tbgenp.c:18:0:
probe.c: In function ‘encode_piece’:
probe.c:821:7: warning: ‘idx’ may be used uninitialized in this function [-Wmaybe-uninitialized]
   idx *= factor[0];
       ^
probe.c: In function ‘decode_pawn’:
probe.c:1640:20: warning: array subscript is above array bounds [-Warray-bounds]
  if (sort[j] > sort[k]) Swap(sort[j], sort[k]);
                    ^
probe.c:1640:43: warning: array subscript is above array bounds [-Warray-bounds]
  if (sort[j] > sort[k]) Swap(sort[j], sort[k]);
                                           ^
tbver.c:96:14: warning: ‘optarg’ redeclared without dllimport attribute: previous dllimport ignored [-Wattributes]
 extern char *optarg;
              ^
In file included from tbver.c:18:0:
probe.c: In function ‘encode_piece’:
probe.c:821:7: warning: ‘idx’ may be used uninitialized in this function [-Wmaybe-uninitialized]
   idx *= factor[0];
       ^
In file included from tbver.c:85:0:
rtbver.c: In function ‘load_wdl’:
rtbver.c:826:7: warning: ‘v1_p’ may be used uninitialized in this function [-Wmaybe-uninitialized]
 printf("WDL_ERROR: idx = %"PRIu64", v2 = %d, v1 = %d\n", idx_p, v2, v1_p);
       ^
In file included from tbver.c:18:0:
probe.c: In function ‘decode_pawn’:
probe.c:1640:20: warning: array subscript is above array bounds [-Warray-bounds]
  if (sort[j] > sort[k]) Swap(sort[j], sort[k]);
                    ^
probe.c:1640:43: warning: array subscript is above array bounds [-Warray-bounds]
  if (sort[j] > sort[k]) Swap(sort[j], sort[k]);
                                           ^
In file included from tbver.c:85:0:
rtbver.c: In function ‘load_dtz’:
rtbver.c:914:36: warning: ‘v1_p’ may be used uninitialized in this function [-Wmaybe-uninitialized]
     table[idx_p] = wdl_to_dtz[v1_p][v2];
                                    ^
tbverp.c:105:14: warning: ‘optarg’ redeclared without dllimport attribute: previous dllimport ignored [-Wattributes]
 extern char *optarg;
              ^
In file included from tbverp.c:20:0:
probe.c: In function ‘encode_piece’:
probe.c:821:7: warning: ‘idx’ may be used uninitialized in this function [-Wmaybe-uninitialized]
   idx *= factor[0];
       ^
In file included from tbverp.c:94:0:
rtbverp.c: In function ‘load_wdl’:
rtbverp.c:1334:34: warning: ‘v1_p’ may be used uninitialized in this function [-Wmaybe-uninitialized]
     table[idx_p] = wdl_matrix[v2][v1_p];
                                  ^
In file included from tbverp.c:20:0:
probe.c: In function ‘decode_pawn’:
probe.c:1640:20: warning: array subscript is above array bounds [-Warray-bounds]
  if (sort[j] > sort[k]) Swap(sort[j], sort[k]);
                    ^
probe.c:1640:43: warning: array subscript is above array bounds [-Warray-bounds]
  if (sort[j] > sort[k]) Swap(sort[j], sort[k]);
                                           ^

I will try to build it with MinGW around weekend and see if it works better.

Thanks,
Kirill

[syzygy]

I finally had the time to try your generator. I could build the 3-4-5 piece tablebases without problems, and all checksums matched yours. However "rtbgen" crashes every time I try to build any 6-piece tablebase - both standalone and via "run.pl". It crashes instantly with no useful diagnostics.

Code: Select all

      0 [main] rtbgen 7224 open_stackdumpfile: Dumping stack trace to rtbgen.exe.stackdump

Did you try the very latest code? It has worked for others, at least with (MingW) gcc-4.7.2.
Does rtbgen.exe.stackdump contain anything?

Another point, GCC printed a bunch or warnings. I humbly suggest that it's perhaps good idea to try and eliminate all warnings, so that the code will give better impression and inspire more confidence.

All array bounds warnings are a bug of gcc-4.8.x. The solution is to add -Wno-array-bounds. The "may be used uninitialized" warnings are all bogus.

Code: Select all

tbgen.c:522:14: warning: ‘optarg’ redeclared without dllimport attribute: previous dllimport ignored [-Wattributes]
 extern char *optarg;

I never saw this warning. I suppose the line could/should be removed.

I will try to build it with MinGW around weekend and see if it works better.

Ah! I never tested with cygwin. It probably can't deal with very large memory allocations.

[Kirill Kryukov]

Did you try the very latest code? It has worked for others, at least with (MingW) gcc-4.7.2.

Yes, of course the latest. It's great to hear that it worked for someone, I will try it with MinGW soon.

Does rtbgen.exe.stackdump contain anything?

Code: Select all

Exception: STATUS_ACCESS_VIOLATION at rip=001801447D5
rax=000006FF31DF0000 rbx=00000001CE210000 rcx=00000001CE210000
rdx=000006FF31DF0000 rsi=00000001CE200030 rdi=00000001CE000010
r8 =00000001CE20FFE2 r9 =0000000000000001 r10=0000000000000000
r11=0000000000000206 r12=0000000000000005 r13=0000000000000007
r14=0000000000000006 r15=000000000022A8A0
rbp=00000001CE000000 rsp=000000000022A5C0
program=K:\egtb\syzygy\rtbgen.exe, pid 11184, thread main
cs=0033 ds=002B es=002B fs=0053 gs=002B ss=002B

If you'd like to debug it, I will be happy to assist in any way I can.

All array bounds warnings are a bug of gcc-4.8.x. The solution is to add -Wno-array-bounds. The "may be used uninitialized" warnings are all bogus.

OK, this is amusing, but if you are sure then it's OK.

Ah! I never tested with cygwin. It probably can't deal with very large memory allocations.

Could be. 64-bit is a very recent addition to cygwin, perhaps something is not sorted yet.