Some questions from a beginner

[The_Algebraist]

I need a little help again

Current status is that the move generation for pawns is working.

Then I started with the slider pieces. I read much about it the last days and decided to try my luck with magic bitboards.

I precomputed the arrays for bishops and rooks as described here: http://chessprogramming.wikispaces.com/ ... for+Magics

Then I used a lookup as suggested here: http://chessprogramming.wikispaces.com/ ... ions-Fancy

When I finally got it working, I tested it with this position I set up for the white bishop on D4:

Code: Select all

 8   r  n  b  .  k  .  .  r
 7   .  p  p  p  .  p  p  p
 6   p  q  .  .  .  n  .  .
 5   .  .  .  .  .  .  .  .
 4   .  .  .  B  .  .  .  .
 3   .  .  b  P  p  P  .  .
 2   P  P  P  .  P  .  P  P
 1   R  N  B  Q  K  .  N  R

     A  B  C  D  E  F  G  H

rnb1k2r/1ppp1ppp/pq3n2/8/3B4/2bPpP2/PPP1P1PP/RNBQK1NR w KQkq - 0 1

and then ran a test as follows:

Code: Select all

printBitboard(bishopAttacks(pos->allPieces(), SQ_D4));

with the result being:

Code: Select all

.  .  .  .  .  .  .  .
x  .  .  .  .  .  .  .
.  x  .  .  .  .  .  .
.  .  x  .  x  .  .  .
.  .  .  .  .  .  .  .
.  .  x  .  x  .  .  .
.  x  .  .  .  x  .  .
x  .  .  .  .  .  x  .

where allPieces returns the following bitboard

Code: Select all

x  x  x  .  x  .  .  x
.  x  x  x  .  x  x  x
x  x  .  .  .  x  .  .
.  .  .  .  .  .  .  .
.  .  .  x  .  .  .  .
.  .  x  x  x  x  .  .
x  x  x  .  x  .  x  x
x  x  x  x  x  .  x  x

For completeness:

Code: Select all

printBitboard(bishopAttacks(pos->allPieces(), SQ_D4) & ~pos->pieces(White));

leads to

Code: Select all

.  .  .  .  .  .  .  .
x  .  .  .  .  .  .  .
.  x  .  .  .  .  .  .
.  .  x  .  x  .  .  .
.  .  .  .  .  .  .  .
.  .  x  .  x  .  .  .
.  .  .  .  .  x  .  .
.  .  .  .  .  .  .  .

These results seem to be wrong. I expect the Squares A7, B2, A1, F2 and G1 to be empty, but have a target on F6.

Now I'm looking for my mistake for quite a while now, but I cannot find it.

Is there a way to test the precomputed magics? I used the function suggested in the cpw to generate them so I assume they are right, but you never know...


For referency (even if taken from the cpw) I paste my relevant code:

Code: Select all

struct SMagic {
   U64  mask;
   U64  magic;
   int  shift;
   U64* ptr;
};

// these are initialized with the precomputed values
extern SMagic bishopTbl[64];
extern SMagic rookTbl[64];

inline U64 bishopAttacks(U64 occ, Square sq) {
   U64* aptr = bishopTbl[sq].ptr;
   occ      &= bishopTbl[sq].mask;
   occ      *= bishopTbl[sq].magic;
   occ     >>= bishopTbl[sq].shift;
   return aptr[occ];
}
 
inline U64 rookAttacks(U64 occ, Square sq) {
   U64* aptr = rookTbl[sq].ptr;
   occ      &= rookTbl[sq].mask;
   occ      *= rookTbl[sq].magic;
   occ     >>= rookTbl[sq].shift;
   return aptr[occ];
}

Could you please have look over my code and give me a hint how I can best find the error?

[Gerd Isenberg]

The attack getting code looks correct, so it seems there goes something wrong with your initialization. Is your square mapping consistent? First check whether occupy-index zero (empty board) leaves the full attack on the otherwise empty board. Then calculate the &*>> bishop occupied index (occ) of the d4-bishop, and put a temporary condition inside the initialization code when storing attacks to that square, occupy-index, to inspect what goes wrong. May be it is a good idea to start sliding piece attacks with something simpler, with same interface, such as Classical Approach, see Hiding the Implementation.

[The_Algebraist]

Hello Gerd,

thank you very much for your reply! I was trying to fix it all day but couldn't get it to work.

With

Code: Select all

printBitboard(bishopAttacks(0, Square(0)));

I get

Code: Select all

.  .  .  .  .  .  .  x
.  .  .  .  .  .  x  .
.  .  .  .  .  x  .  .
.  .  .  .  x  .  .  .
.  .  .  x  .  .  .  .
.  .  x  .  .  .  .  .
.  x  .  .  .  .  .  .
.  .  .  .  .  .  .  .

which is correct I think.



I can inspect the index, but I don't get much information from that (it's 484 in this case), because I precomputed the arrays using the method suggested in the wiki:
http://chessprogramming.wikispaces.com/ ... for+Magics

I guess this is where something went wrong.

The bishop magics look like this:

Is there any method I can verify these?

Maybe I really try another approach.
Is there an approach that is good to implement but not too slow?

How is the the classical variant compared to the hyperbola quintessence?


Edit:

My square-representation is just an enum:

Code: Select all

enum Square {
    // Little endian rank-file (LERF) mapping
    SQ_A1, SQ_B1, SQ_C1, SQ_D1, SQ_E1, SQ_F1, SQ_G1, SQ_H1,
    SQ_A2, SQ_B2, SQ_C2, SQ_D2, SQ_E2, SQ_F2, SQ_G2, SQ_H2,
    SQ_A3, SQ_B3, SQ_C3, SQ_D3, SQ_E3, SQ_F3, SQ_G3, SQ_H3,
    SQ_A4, SQ_B4, SQ_C4, SQ_D4, SQ_E4, SQ_F4, SQ_G4, SQ_H4,
    SQ_A5, SQ_B5, SQ_C5, SQ_D5, SQ_E5, SQ_F5, SQ_G5, SQ_H5,
    SQ_A6, SQ_B6, SQ_C6, SQ_D6, SQ_E6, SQ_F6, SQ_G6, SQ_H6,
    SQ_A7, SQ_B7, SQ_C7, SQ_D7, SQ_E7, SQ_F7, SQ_G7, SQ_H7,
    SQ_A8, SQ_B8, SQ_C8, SQ_D8, SQ_E8, SQ_F8, SQ_G8, SQ_H8,

    SQ_None = 64
};

[kbhearn]

the classical approach gerd linked is not so slow as to be the biggest problem your beginning program will have and since it shares the same function prototype (a square and an occupied bitboard as parameters), at the point when you have to worry about the speedup changing it just involves changing that function, not the rest of the code.

[Sven]

Is there a way to test the precomputed magics? [...]

Code: Select all

inline U64 bishopAttacks(U64 occ, Square sq) {
   U64* aptr = bishopTbl[sq].ptr;
   occ      &= bishopTbl[sq].mask;
   occ      *= bishopTbl[sq].magic;
   occ     >>= bishopTbl[sq].shift;
   return aptr[occ];
}
 
inline U64 rookAttacks(U64 occ, Square sq) {
   U64* aptr = rookTbl[sq].ptr;
   occ      &= rookTbl[sq].mask;
   occ      *= rookTbl[sq].magic;
   occ     >>= rookTbl[sq].shift;
   return aptr[occ];
}

Could you please have look over my code and give me a hint how I can best find the error?

I would add assertions into the code above to ensure that all involved values are inside the allowed ranges. Test 'sq', 'shift', 'aptr', and 'occ' (after shifting) at least. In most cases 'shift' should be between (64-5) and (64-9) for bishops and between (64-10) and (64-12) for rooks.

This is kind of a late test of your initialization code so of course it would be even better to put assertions there directly.

The generated magic numbers themselves will most probably be correct since you have obviously written generation code similar to the one described on the CPW page to which you linked above, and that code contains a test whether the magic number "works" for the given square (in fact this is part of the algorithm). But your other code that maintains all the data could still have a problem.

[Sven]

My current guess is that there is indeed something wrong with your generated magic numbers. This part of the original generator code from Tord Romstad (in function find_magic()):

Code: Select all

    for(i = 0; i < 4096; i++) used[i] = 0ULL;
    for(i = 0, fail = 0; !fail && i < (1 << n); i++) {
      j = transform(b[i], magic, m);
      if(used[j] == 0ULL) used[j] = a[i];
      else if(used[j] != a[i]) fail = 1;
    }
    if(!fail) return magic;

does the check that I mentioned above. If you have not implemented such a check, or if there is some error in your implementation of it, then it can cause the effect that you see. The check ensures that the mapping of each occupancy to the corresponding "occupancy index" ("j", or "occ" in the using code) is unique.

[Gerd Isenberg]

OK, occupancy index 0 should be fine for all squares, attacks on the otherwise empty board. But 484 contains the wrong attack for that occupancy from d4! Why? What happens during initialization?

Code: Select all

uint64 find_magic(int sq, int m, int bishop) {
  uint64 mask, b[4096], a[4096], used[4096], magic;
  int i, j, k, n, fail;
 
  mask = bishop? bmask(sq) : rmask(sq);
  n = count_1s(mask);
 
  for(i = 0; i < (1 << n); i++) {
    b[i] = index_to_uint64(i, n, mask);
    a[i] = bishop? batt(sq, b[i]) : ratt(sq, b[i]);
  }
  for(k = 0; k < 100000000; k++) {
    magic = random_uint64_fewbits();
    if(count_1s((mask * magic) & 0xFF00000000000000ULL) < 6) continue;
    for(i = 0; i < 4096; i++) used[i] = 0ULL;
    for(i = 0, fail = 0; !fail && i < (1 << n); i++) {
      j = transform(b[i], magic, m);
      if(used[j] == 0ULL) used[j] = a[i];
      else if(used[j] != a[i]) fail = 1;
#if _DEBUG
      if ( !fail && bishop && sq == SQ_D4 && j == 484 )
         printBitboard(a[i]); // is this the right attack set?
#endif
    }
    if(!fail) return magic;
  }
  printf("***Failed***\n");
  return 0ULL;
}

[The_Algebraist]

Thank you for your great help Gerd, Sven and Kevin!

Gerd: I can't exactly tell what happens during initialization, because I precomputed the magics, and when I recompute them, they change.

That is another learning point I took from this, when I implement the magics the next time, I won't precompute them but generate them on startup. Then I can just have a constant seed and will end up with the same magics every time.


I followed the advice and implemented the classical approach for now. It works very good, but I still need to implement castling until I can start Perft-testing and see how it does in respect to performance .

But these magic bitboards keep sticking in my head!

I thought about it some more and will definitely come back and try to implement it again, better sooner than later.

One question I came up with when thinking about it:

In the article: http://chessprogramming.wikispaces.com/ ... for+Magics
I think I understand most of the algorithm that is suggested. But one thing is unclear to me, what is the point of this function:

Code: Select all

uint64 index_to_uint64(int index, int bits, uint64 m) {
  int i, j;
  uint64 result = 0ULL;
  for(i = 0; i < bits; i++) {
    j = pop_1st_bit(&m);
    if(index & (1 << i)) result |= (1ULL << j);
  }
  return result;
}

It is called here:

Code: Select all

uint64 find_magic(int sq, int m, int bishop) {
  uint64 mask, b[4096], a[4096], used[4096], magic;
  int i, j, k, n, fail;
 
  mask = bishop? bmask(sq) : rmask(sq);
  n = count_1s(mask);
 
  for(i = 0; i < (1 << n); i++) {
    b[i] = index_to_uint64(i, n, mask);
    a[i] = bishop? batt(sq, b[i]) : ratt(sq, b[i]);
  }
  for(k = 0; k < 100000000; k++) {
    magic = random_uint64_fewbits();
    if(count_1s((mask * magic) & 0xFF00000000000000ULL) < 6) continue;
    for(i = 0; i < 4096; i++) used[i] = 0ULL;
    for(i = 0, fail = 0; !fail && i < (1 << n); i++) {
      j = transform(b[i], magic, m);
      if(used[j] == 0ULL) used[j] = a[i];
      else if(used[j] != a[i]) fail = 1;
    }
    if(!fail) return magic;
  }
  printf("***Failed***\n");
  return 0ULL;
}

and I assume describes the blockers. Anyways I don't quite understand how the function 'index_to_uint64' is connected with the computation of the blockers or the rest of the algorithm?

[Sven]

One question I came up with when thinking about it:

In the article: http://chessprogramming.wikispaces.com/ ... for+Magics
I think I understand most of the algorithm that is suggested. But one thing is unclear to me, what is the point of this function:

Code: Select all

uint64 index_to_uint64(int index, int bits, uint64 m) {
  int i, j;
  uint64 result = 0ULL;
  for(i = 0; i < bits; i++) {
    j = pop_1st_bit(&m);
    if(index & (1 << i)) result |= (1ULL << j);
  }
  return result;
}

It is called here:

Code: Select all

uint64 find_magic(int sq, int m, int bishop) {
  uint64 mask, b[4096], a[4096], used[4096], magic;
  int i, j, k, n, fail;
 
  mask = bishop? bmask(sq) : rmask(sq);
  n = count_1s(mask);
 
  for(i = 0; i < (1 << n); i++) {
    b[i] = index_to_uint64(i, n, mask);
    a[i] = bishop? batt(sq, b[i]) : ratt(sq, b[i]);
  }
  for(k = 0; k < 100000000; k++) {
    magic = random_uint64_fewbits();
    if(count_1s((mask * magic) & 0xFF00000000000000ULL) < 6) continue;
    for(i = 0; i < 4096; i++) used[i] = 0ULL;
    for(i = 0, fail = 0; !fail && i < (1 << n); i++) {
      j = transform(b[i], magic, m);
      if(used[j] == 0ULL) used[j] = a[i];
      else if(used[j] != a[i]) fail = 1;
    }
    if(!fail) return magic;
  }
  printf("***Failed***\n");
  return 0ULL;
}

and I assume describes the blockers. Anyways I don't quite understand how the function 'index_to_uint64' is connected with the computation of the blockers or the rest of the algorithm?

index_to_uint64() calculates the "blockers". For a given square and piece type (bishop/rook) there may be two or more different occupancies sharing the same set of blockers. E.g. a bishop on d1 does not attack g4 if there is a piece on e2, regardless whether f3 is occupied as well or not. Same blockers must imply same attack set. The check that is built-in into the generation algorithm verifies that the current magic number fulfills this last condition by checking that the transformation (i.e., magic multiplication + shift) of the obtained "blockers" bitboard returns a unique attack set index. If this is not the case then the current magic number is rejected.

[Gerd Isenberg]

Ok, first there are n = 5..12 relevant occupancy bits pers square for bishops / rooks, thus 2^n (1 << n), max 4096 possible relevant occupancy states. The first step per square in Tord's find magic routine is to initialize the b[] and a[] arrays with all possible occupancies (b[]) and slowly computed attacks (a[], batt, ratt) by square and occupancy b.

Code: Select all

  mask = bishop? bmask(sq) : rmask(sq);
  n = count_1s(mask);
 
  for(i = 0; i < (1 << n); i++) {
    b[i] = index_to_uint64(i, n, mask);
    a[i] = bishop? batt(sq, b[i]) : ratt(sq, b[i]);
  }

index_to_uint64 (may be a better name is index_to_occupancy) is responsible to compute all occupancies for that square by mask, and n (popcnt of mask), and an enumerator index, from 0 to 2^n -1.

Code: Select all

uint64 index_to_uint64(int index, int bits, uint64 m) {
  int i, j;
  uint64 result = 0ULL;
  for(i = 0; i < bits; i++) {
    j = pop_1st_bit(&m);
    if(index & (1 << i)) result |= (1ULL << j);
  }
  return result;
}

The routine starts with occupancy result of zero.

Code: Select all

  uint64 result = 0ULL;

And loops over all possible occupancy bits (5 ... 12) ...

Code: Select all

  for(i = 0; i < bits; i++) {

and gets the next set bit-index from the mask (bitscan forward with reset)

Code: Select all

    j = pop_1st_bit(&m);

It then sets the corresponding bit (2^j, 1 << j) inside the result, if the index (0..2^n-1, max 4095) has bit i set.

Code: Select all

    if(index & (1 << i)) result |= (1ULL << j);

This is how all possible occupancies are computed by all possible indices. Note that there are far less attack sets than occupancies, since bits behind the first blocker don't affect the attacks. Thus, while all b are different, many a are equal for different i.

===

The next step is to find a magic factor, chosen randomly by trial and error (up to 100000000 trials).

Code: Select all

  for(k = 0; k < 100000000; k++) {
    magic = random_uint64_fewbits();

The next line is some empirical optimization (I guess), since it turned out that "good" magics to try require at least six bit set in the upper byte of a 64-bit product:

Code: Select all

    if(count_1s((mask * magic) & 0xFF00000000000000ULL) < 6) continue;

The used[] is cleared to check for collisions ...

Code: Select all

    for(i = 0; i < 4096; i++) used[i] = 0ULL;

Looping over all occupancies again so long there is no destructive collission ...

Code: Select all

    for(i = 0, fail = 0; !fail && i < (1 << n); i++) {

calculate the occupancy index, using precalculated b[] again (see above) ...

Code: Select all

      j = transform(b[i], magic, m);

if not used yet, store attacks

Code: Select all

      if(used[j] == 0ULL) used[j] = a[i];

if used, we have a collission, but may be a (desired) constructive one, if different occupancies share same attacks due to different occupancy bits "behind" first blocker. That is why we store attacks in used, and magic only fails, if attacks are different too.

Code: Select all

      else if(used[j] != a[i]) fail = 1;

otherwise we have found a destructive collission free magic factor ...

Code: Select all

    if(!fail) return magic;

And yes, you can compute the magics on the fly while startup ...