7-men Syzygy attempt

Discussion of chess software programming and technical issues.

Moderators: hgm, Rebel, chrisw

noobpwnftw
Posts: 560
Joined: Sun Nov 08, 2015 11:10 pm

Re: 7-men Syzygy attempt

Post by noobpwnftw »

I found a problem(bug?) after raising the stats limit:
Although later we need to raise the map to 16-bits for large DTZ compression, in the current pawn code the MAX_PLY is 768, so this array here written with [(MAX_PLY - 1 - DRAW_RULE) / 2] is overflowed.

It does not seem to affect the correctness(confirm?).
Dann Corbit
Posts: 12534
Joined: Wed Mar 08, 2006 8:57 pm
Location: Redmond, WA USA

Re: 7-men Syzygy attempt

Post by Dann Corbit »

noobpwnftw wrote:I found a problem(bug?) after raising the stats limit:
Although later we need to raise the map to 16-bits for large DTZ compression, in the current pawn code the MAX_PLY is 768, so this array here written with [(MAX_PLY - 1 - DRAW_RULE) / 2] is overflowed.
. Other
It does not seem to affect the correctness(confirm?).
Expand the array. Otherwise undefined behavior
Taking ideas is not a vice, it is a virtue. We have another word for this. It is called learning.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.
syzygy
Posts: 5555
Joined: Tue Feb 28, 2012 11:56 pm

Re: 7-men Syzygy attempt

Post by syzygy »

noobpwnftw wrote:I found a problem(bug?) after raising the stats limit:
Although later we need to raise the map to 16-bits for large DTZ compression, in the current pawn code the MAX_PLY is 768, so this array here written with [(MAX_PLY - 1 - DRAW_RULE) / 2] is overflowed.

It does not seem to affect the correctness(confirm?).
Yes this is part of the code that has to be adapted to deal with 16-bit values. The problem starts when (dtz-101)/2 exceeds 255 (actually a bit earlier). This should not affect the generation process and statistics are probably collected correctly.
koedem
Posts: 105
Joined: Fri Mar 18, 2016 10:45 pm

Re: 7-men Syzygy attempt

Post by koedem »

I realized only now, you generated a complete material balance in 30 seconds only? I have to imagine the ones with pawns take much more RAM because you need to cache the possible promotions so those might be more tricky. But nontheless I am surprised that the pawnless ones go so smooth.

Which hardware configuration did you use? Also, in case one needs a lot of RAM already for the pawnless ones, do you think you could upload some of them when they're finished? (maybe via some torrent or so)
syzygy
Posts: 5555
Joined: Tue Feb 28, 2012 11:56 pm

Re: 7-men Syzygy attempt

Post by syzygy »

No, the 33 seconds is only for the "resetting cursed losses" pass.
duncan
Posts: 12038
Joined: Mon Jul 07, 2008 10:50 pm

Re: 7-men Syzygy attempt

Post by duncan »

noobpwnftw wrote: With your new 16-bit compression code available, I'm pretty confident that full set of 7-men Syzygy will be available soon.

I already have the necessary hardware(both steps in physical RAM without --disk) and will provide hosting for all the resulting files.
great achievement if you do this. Thanks.
User avatar
Nordlandia
Posts: 2821
Joined: Fri Sep 25, 2015 9:38 pm
Location: Sortland, Norway

Re: 7-men Syzygy attempt

Post by Nordlandia »

Bojun Guo: have you checked if probing 7-man inflict larger speed penalty than usual. Probing 6-men cause slight engine speed reduction, not sure how much force 7-man add on speed.

Maybe too early to say.
noobpwnftw
Posts: 560
Joined: Sun Nov 08, 2015 11:10 pm

Re: 7-men Syzygy attempt

Post by noobpwnftw »

Currently all work is about making sure the generator can produce correct results and the file format can hold them, since I don't want to find something broken in the middle of the building campaign...

A quick example is this, the second part overflowed when building KQQQQPvK, luckily it caused crash because MSB before expansion was 1 so the index became negative, otherwise it would just remain silent with a wrong index.

If anyone interested, please also help with checking(take my fork as reference) for such overflows when piece count, index shift range and pawn mask has increased in the 7-men case.
Dann Corbit
Posts: 12534
Joined: Wed Mar 08, 2006 8:57 pm
Location: Redmond, WA USA

Re: 7-men Syzygy attempt

Post by Dann Corbit »

noobpwnftw wrote:Currently all work is about making sure the generator can produce correct results and the file format can hold them, since I don't want to find something broken in the middle of the building campaign...

A quick example is this, the second part overflowed when building KQQQQPvK, luckily it caused crash because MSB before expansion was 1 so the index became negative, otherwise it would just remain silent with a wrong index.

If anyone interested, please also help with checking(take my fork as reference) for such overflows when piece count, index shift range and pawn mask has increased in the 7-men case.
Odd errors trying to use clang.
file defs.h has this:
typedef unsigned long long bitboard;

And yet I get these warnings/errors:

Code: Select all

dcorbit@dcorbit MINGW64 /f/project/dcorbit/tb-7-men/src
$ make
make[1]: Entering directory 'F:/project/dcorbit/tb-7-men/src'
clang -pthread  -O3 -march=native -pipe -D_GNU_SOURCE -Wall -std=c99 -fsanitize=undefined -Wno-array-bounds -flto -fprofile-use -lgcov -c tbgen.c -DHYPER -DUSE_POPCNT -DREGULAR -o objsr/tbgen.o
clang.exe: warning: -lgcov: 'linker' input unused [-Wunused-command-line-argument]
error: Could not read profile default.profdata: No such file or directory
In file included from tbgen.c:16:
In file included from ./board.h:33:
./hyper.h:47:7: warning: implicit declaration of function '_mm_cvtsi64x_si128' is invalid in C99 [-Wimplicit-function-declaration]
  o = _mm_cvtsi64x_si128(occ);
      ^
./hyper.h:47:5: error: assigning to '__m128i' (vector of 2 'long long' values) from incompatible type 'int'
  o = _mm_cvtsi64x_si128(occ);
    ^ ~~~~~~~~~~~~~~~~~~~~~~~
In file included from tbgen.c:18:
In file included from ./board.c:9:
./hyper.c:139:10: error: assigning to '__m128i' (vector of 2 'long long' values) from incompatible type 'int'
    xmm1 = _mm_cvtsi64x_si128(bb);
         ^ ~~~~~~~~~~~~~~~~~~~~~~
./hyper.c:140:10: error: assigning to '__m128i' (vector of 2 'long long' values) from incompatible type 'int'
    xmm2 = _mm_cvtsi64x_si128(bb2);
         ^ ~~~~~~~~~~~~~~~~~~~~~~~
./hyper.c:142:10: error: assigning to '__m128i' (vector of 2 'long long' values) from incompatible type 'int'
    xmm1 = _mm_cvtsi64x_si128(bit[sq]);
         ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~
./hyper.c:163:8: error: assigning to '__m128i' (vector of 2 'long long' values) from incompatible type 'int'
  xmm1 = _mm_cvtsi64x_si128(0x0001020304050607);
       ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
./hyper.c:164:8: error: assigning to '__m128i' (vector of 2 'long long' values) from incompatible type 'int'
  xmm2 = _mm_cvtsi64x_si128(0x08090a0b0c0d0e0f);
       ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from tbgen.c:89:
./rtbgen.c:128:5: error: 'asm goto' constructs are not supported yet
    jump_bit_set(bb, sq, lab);
    ^
./generics.c:218:11: note: expanded from macro 'jump_bit_set'
  __asm__ goto ("bt %1, %0; jc %l[lab]" : : "r" (x), "r" ((long64)(y)) : : lab);
          ^
1 warning and 8 errors generated.
make[1]: *** [Makefile.general:30: tbgen.o] Error 1
make[1]: Leaving directory 'F:/project/dcorbit/tb-7-men/src'
make: *** [Makefile:34: rtbgen] Error 2
Taking ideas is not a vice, it is a virtue. We have another word for this. It is called learning.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.
noobpwnftw
Posts: 560
Joined: Sun Nov 08, 2015 11:10 pm

Re: 7-men Syzygy attempt

Post by noobpwnftw »

koedem wrote: Which hardware configuration did you use? Also, in case one needs a lot of RAM already for the pawnless ones, do you think you could upload some of them when they're finished? (maybe via some torrent or so)
I used a 4-way Xeon with some physical RAM more than 1TB to build them. Actually the pawnful ones require less memory because they are built per file.
I have hosted all 3-6 files in my FTP and there is a folder for some test files of 7 piece TBs. Currently the my further building is on hold because there will be a DTZ format change in order to hold all cases in 7 piece TBs, it may or may not require a rebuild of these files.

Don't take anything for granted, these files may not be correct at all.

P.S. on hosting, you can use whatever download tools you want, there is no limit on bandwidth usage nor will I null route anyone.
I had a very bad experience even downloading single threaded somewhere else and I can't use torrent.