The mailbox trials

Discussion of chess software programming and technical issues.

Moderators: hgm, Rebel, chrisw

Mike Sherwin
Posts: 860
Joined: Fri Aug 21, 2020 1:25 am
Location: Planet Earth, Sol system
Full name: Michael J Sherwin

Re: The mailbox trials

Post by Mike Sherwin »

hgm wrote: Mon Mar 29, 2021 7:13 pm Sorry! It appears I uploaded a file with the same name from another directory. :oops: I have now uploaded the correct one.
1 -16 725 0 e2a6 b4c3 b2c3 e6d5
2 -16 9497 0 e2a6 b4c3 b2c3 e6d5
3 -27 12797 0 e2a6 e6d5 g2h3 b4c3 d2c3 d5e4
4 -27 71479 0 e2a6 e6d5 g2h3 b4c3 d2c3 d5e4
5 -37 149317 0 e2a6 e6d5 c3d5 b6d5 e4d5 e7e5
6 -32 399081 0 e2a6 e6d5 c3d5 b6d5 a6b7 a8b8 b7d5 f6d5 e5f7 e7f7 e4d5 g7b2 f3f7 e8f7
7 -32 838855 0 e2a6 e6d5 c3d5 b6d5 a6b7 a8b8 b7d5 f6d5 e5f7 e7f7 e4d5 g7b2 f3f7 e8f7
8 -32 4710223 0 e2a6 e6d5 c3d5 b6d5 a6b7 a8b8 b7d5 f6d5 e5f7 e7f7 e4d5 g7b2 f3f7 e8f7
t = 0.485 sec
4710223 nodes (9.7 Mnps)
4409258 QS (93.6%)
905777 evals (20.5%)
350015 stand-pats (7.9%)
1239841 leaves (28.1%)
4134586 move gens
captures: 94.1%

Changes to compile with MSVS 2019 Community
#include <stdlib.h>
#pragma warning(disable : 4996)
char KIWIPETE[] = "r3k2r/p1ppqpb1/bn2pnp1/3PN3/1p2P3/2N2Q1p/PPPBBPPP/R3K2R w KQkq - 0 1";
char FIDE[] = "rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - 0 1";
User avatar
hgm
Posts: 27788
Joined: Fri Mar 10, 2006 10:06 am
Location: Amsterdam
Full name: H G Muller

Re: The mailbox trials

Post by hgm »

Every time I try to implement staged move generation I come up with something that disgusts me. But I still cannot find anything more elegant. What I have inserted now at the beginning of the move loop is this:

Code: Select all

first = msp;               // initialize empty move list
nextVictim = stm + KING;   // opponent King (stm is already flipped here!)

for(curMove=first; 1; curMove++) {     // (infinite) loop over moves
  // move source
  if(curMove >= msp) {                 // move list is empty, generate next badge
    if(nextVictim >= stm) {            // we are still in capture stage
      do {
        int to = location[nextVictim]; // consider this victim
        if(to == CAPTURED) continue;   // not present, try next
        Capture(to);                   // generate captures on it
        if(curMove >= msp) continue;   // was not attacked, try next
        if(nextVictim == stm + KING) { // attacked piece is King
          alpha = INF; goto cutoff;    // abort move generation due to King capture
        }
        groupValue = mvv[nextVictim];  // value of the attacked piece
        while(mvv[--nextVictim] == groupValue) { // look for equal-valued others
          to = location[nextVictim];
          if(to == CAPTURED) continue; // not present, next
          Capture(to);                 // add captures on that too
        }
        goto havemove;                 // process next batch of captures
      } while(--nextVictim >= stm);    // keep looking for attacked piece
    } else if(nextVictim < 0) break;   // nothing left to generate, terminate move loop
    GenNoncapts();                     // generate non-captures
    if(curMove >= msp) break;          // no noncaptures, terminate move loop
    nextVictim = -1;                   // signal that we are done
  }
 havemove:
  // sort section (for sorting batch by LVA)
Reaching the end of the move list no longer terminates the loop, but results in an attempt to generate a new batch of moves (and only terminates the loop if that doesn't succeed). A 'stage' consists of all captures on pieces of the same value, or all non-captures; the variable nextVictim indicates the stage that has to be done next. After all victims are exhausted non-captures are generated, and nextVictim is set to the artificial value -1 to indicate this has been done.

What I dislike about the code is that it needs a goto to skip noncapture generation when it found captures, and that the function Capture() (which generates the captures to a given square) has to be called in two places.
Mike Sherwin
Posts: 860
Joined: Fri Aug 21, 2020 1:25 am
Location: Planet Earth, Sol system
Full name: Michael J Sherwin

Re: The mailbox trials

Post by Mike Sherwin »

hgm wrote: Tue Mar 30, 2021 10:55 am Every time I try to implement staged move generation I come up with something that disgusts me. But I still cannot find anything more elegant. What I have inserted now at the beginning of the move loop is this:

Code: Select all

first = msp;               // initialize empty move list
nextVictim = stm + KING;   // opponent King (stm is already flipped here!)

for(curMove=first; 1; curMove++) {     // (infinite) loop over moves
  // move source
  if(curMove >= msp) {                 // move list is empty, generate next badge
    if(nextVictim >= stm) {            // we are still in capture stage
      do {
        int to = location[nextVictim]; // consider this victim
        if(to == CAPTURED) continue;   // not present, try next
        Capture(to);                   // generate captures on it
        if(curMove >= msp) continue;   // was not attacked, try next
        if(nextVictim == stm + KING) { // attacked piece is King
          alpha = INF; goto cutoff;    // abort move generation due to King capture
        }
        groupValue = mvv[nextVictim];  // value of the attacked piece
        while(mvv[--nextVictim] == groupValue) { // look for equal-valued others
          to = location[nextVictim];
          if(to == CAPTURED) continue; // not present, next
          Capture(to);                 // add captures on that too
        }
        goto havemove;                 // process next batch of captures
      } while(--nextVictim >= stm);    // keep looking for attacked piece
    } else if(nextVictim < 0) break;   // nothing left to generate, terminate move loop
    GenNoncapts();                     // generate non-captures
    if(curMove >= msp) break;          // no noncaptures, terminate move loop
    nextVictim = -1;                   // signal that we are done
  }
 havemove:
  // sort section (for sorting batch by LVA)
Reaching the end of the move list no longer terminates the loop, but results in an attempt to generate a new batch of moves (and only terminates the loop if that doesn't succeed). A 'stage' consists of all captures on pieces of the same value, or all non-captures; the variable nextVictim indicates the stage that has to be done next. After all victims are exhausted non-captures are generated, and nextVictim is set to the artificial value -1 to indicate this has been done.

What I dislike about the code is that it needs a goto to skip noncapture generation when it found captures, and that the function Capture() (which generates the captures to a given square) has to be called in two places.
I do not know what your criteria for disgust or eloquence is so all I can do is suggest an alternative which would be:

stage = FIRSTSTAGE;

switch (stage) {
....case FIRSTSTAGE:
........SomeCode();
........if (!done) break;
........stage = SECONDSTAGE;
........break;
....case SECONDSTAGE:
.
.
.
Or something like that.
User avatar
hgm
Posts: 27788
Joined: Fri Mar 10, 2006 10:06 am
Location: Amsterdam
Full name: H G Muller

Re: The mailbox trials

Post by hgm »

I guess that would be cleaner. But it would also bloat the code quite a bit, and my first 5 stages would be capture-King, capture-Queen, capture-Rook, capture-minor, capture-Pawn, which would basically all have identical code, except applied to another group of victims in the opponent piece list.

Code: Select all

case captMinor:
  for(victim=stm+11; victim>=stm+8; victim--) {
      int to = location[victim];
      if(to == CAPTURED) continue;
      Capture(to);
  }
  stage = captPawn;
  if(msp > curMove) break;
case captPawn:
Hmm, good suggestion, I suppose. It does look a lot less contrived. If I am smart I can eliminate part of the loop overhead by identifying victim with stage:

Code: Select all

switch(victim) {
...
case 11: // Bishops index in piece list is 10 and 11
  do {
    int to = location[victim + stm];
    if(to == CAPTURED) continue;
    Capture(to);
  } while(--victim >= 8); // up to Knights (which are 8 and 9)
  // victim will automatically end up at 7, which is next stage
  if(msp > curMove) break;
case 7: // Pawn
  ...
case -1:
  GenNoncapts();
  victim = -2;
case -2:
  goto cutoff; // cannot use break to break out of loop in switch. (C design flaw?)
}
I suppose I could put the code for the capture cases in a function (to be inlined), which returns whether moves were generated:

Code: Select all

int CaptureGroup(UndoInfo *u, int lowest)
{
  do {
    int to = location[u->nextVictim + stm];
    if(to == CAPTURED) continue;
    Capture(to);
  } while(--u->nextVictim >= lowest);
  return (msp > curMove);
}
and inline (or perhaps explicitly write out, as it now occurs only once) Capture(to). The move source then would look:

Code: Select all

if(undo.nextVictim == -2) break; // breaks out of move loop
switch(undo.nextVictim) {
  case 15: if(CaptureGroup(&undo, 15)) { alpha = INF; goto cutoff; } // abort on King capture
  case 14: if(CaptureGroup(&undo, 14)) break; // Queen
  case 13: if(CaptureGroup(&undo, 12)) break; // Rooks
  case 11: if(CaptureGroup(&undo,  8)) break; // minors
  case  7: if(CaptureGroup(&undo, 0)) break; // Pawns
  case -1: if(GenNoncapts()) { undo.nextVictim = -2; break; }
}
Thanks for the suggestion!
User avatar
hgm
Posts: 27788
Joined: Fri Mar 10, 2006 10:06 am
Location: Amsterdam
Full name: H G Muller

Re: The mailbox trials

Post by hgm »

OK, I did the conversion to move generation by victim. As expected this is less efficient. But the difference with the per-attacker generation is not nearly as much as I feared: only about 5%:

Code: Select all

 1  -16       490 0 e2a6 b4c3 b2c3 e6d5
 2  -16      7660 0 e2a6 b4c3 b2c3 e6d5
 3  -27     11670 0 e2a6 e6d5 g2h3 b4c3 d2c3 d5e4
 4  -27     78170 0 e2a6 e6d5 g2h3 b4c3 d2c3 d5e4
 5  -37    174976 0 e2a6 e6d5 c3d5 b6d5 e4d5 e7e5
 6  -32    465458 0 e2a6 e6d5 c3d5 b6d5 a6b7 a8b8 b7d5 f6d5 e5f7 e7f7 e4d5 g7b2 f3f7 e8f7
 7  -32   1040493 0 e2a6 e6d5 c3d5 b6d5 a6b7 a8b8 b7d5 f6d5 e5f7 e7f7 e4d5 g7b2 f3f7 e8f7
 8  -32   4770530 0 e2a6 e6d5 c3d5 b6d5 a6b7 a8b8 b7d5 f6d5 e5f7 e7f7 e4d5 g7b2 f3f7 e8f7
t =  0.840 sec
   4770530 nodes (5.7 Mnps)
   4423365 QS (92.7%)
    907648 evals (20.5%)
    340650 stand-pats (7.7%)
   1505525 leaves (34.0%)
         0 move gens
captures: 93.4%
The node counts are not exactly the same. But this cannot be expected, as the captures can now be generated in a different order. They are still searched in MVV/LVA order, but this is not unique. And I guess the previous version was actually buggy in this respect, as I now discovered it was using a different sort key for Bishop and Knight victims. The current code treats the minors as a single value group, as it was supposed to do, because it is not dependent anymore on the value of any sort key for that: the MVV ordering has become an intrinsic property of the generation method. But the tree statistics is very similar, and what the exact tree is should not affect the nps much.

Strangely inlining the CaptureGroup routine slowed it down.

Even within the framework of from-scratch generation, there are some obvious inefficiencies here. E.g. CaptureGroup() is also applied to the King, for detecting its capture, and there are more efficient methods for detecting whether a move puts your King in check. Like testing for a pre-existing check in the parent node, and then for non-King moves just detect whether that still exists, and whether the moved piece delivered or discovered a check, rather than just looking in 8 directions and testing all Knights. These methods are actually incremental. So I will not try such improvements now, but work to the goal of keeping track of all attacks incrementally, not just those on a King.

As discussed in the early postings of this blog, this will be done through an attack map that indicates for each piece the set of its attackers in a single machine word, and a 'summary word' that holds the number of attacks on each piece as packed counters. So that move generation is reduced to just extracting the set bits in attack map, and looping over the possible victims can be done by extracting bits from the summary word. As the extraction of attackers will happen in LVA order, and those of victims in MVV order, move sorting is no longer necessary. That also removes the need to store the captures in a move list.

We will thus have to refactor the move loop and the staged move source completely. That is, for the non-captures this can stay, but all the stages for generating captures will be replaced by a single loop, which not only finds the moves, but immediately searches them. Like:

Code: Select all

uint64_t todo = summaryWord;
while((groupMask = groupTable[NextBit(todo)])) { // for all attacked pieces
  uint64_t groupMask &= summaryWord; // leave only pieces in the value group
  uint64_t groupAttacks = 0;
  int victimSP = 0;
  while((victim = victimTable[NextBit(groupMask)])) { // for all attacked victims in value group
    groupAttacks = 2*groupAttacks + attackers[victim]; // merge the attackers sets from the attack map
    victimStack[victimSP++] = victim; // remember how victims correspond with bits in the merged set
  }
  while((capture = NextBit(groupAttacks))) { // this will now extract in MVV/LVA order
    undo.piece = stm + (capture >> 2);
    undo.victim = victimStack[capture & 3];
    undo.from = location[undo.piece];
    undo.to = location[undo.victim];
    if(SearchMove(&undo)) goto cutoff; // returns TRUE for beta cutoff
  }
}
The speed of this method should come from the fact that it spends zero time on things that do not produce a result. The from-scratch per-victim generation does a test for each possible source of attack (8 directions + 2 Knights), even for pieces that are not attacked at all. But here unattacked pieces are skipped, either together with their entire value group (if none of the pieces in the group was attacked), or in the loop over victims within a group. Ony the attackers set of actually attacked pieces are merged into groupAttacks, from which only actual captures are extracted.

A new routine SearchMove() is introduced, which combines the functions of MakeMove(), Search(), UnMake() and the minimaxing of the score. This requires the best score ('alpha' in our fail-hard scheme) to be put in the undo struct as well.
User avatar
hgm
Posts: 27788
Joined: Fri Mar 10, 2006 10:06 am
Location: Amsterdam
Full name: H G Muller

Re: The mailbox trials

Post by hgm »

32 or 64 bits?

To count the number of attacks on a piece needs 4 bits, as a piece can potentially have 9 attackers (2 P, 1B, 2N, 2R, 1Q, 1K). For all 16 possible victims this requires 64 bits. So the counters of the attack summary could be nicely packed into a single 64-bit int. The attackers sets in the attack map only need 16 bits. But to indicate the combined attacks on a value group needs a multiple of that. With 64-bit attackers sets we could spread out the 16 bits indicating the attackers over the word, in a 0x1111111111111111ull ('COMB') pattern, in order of attacker value. It is then possible to merge 4 sets (as set1 + 2*set2 + 4*set3 + 8*set4) without disturbing the LVA order in which the attackers would extract.

Unfortunately the largest value group has 8 victims: the Pawns. So even with 64-bit attackers sets this group would have to be split into two. Treating them as the other value groups would then treat piece x Pawn captures of the first group before PxP captures of the other (or QxP of one before NxP of the other), which violates the LVA part of MVV/LVA. This could be solved by exctracting from both value groups at once, and then on a case by case basis decide which of the two capture candidates is next in line. Note that this is not needed in the Pawn phase: we can safely extract all PxP captures from group 1, and then all PxP from group 2, as no capture on a Pawn could have higher priority than PxP. The special treatment would only be needed for piece x Pawn, to make sure the lowest (non-Pawn) piece is treated first.

This brings me to the issue: would it be very harmful to split everything into a Pawn and a piece part, so we would not need 64-bit integers? For updating the attack map this would make very little difference. Changes there are caused by removing and adding the attacks by the moved piece, and discovery and blocking of sliders. In all these cases the piece making the attacks is a given, and you can ignore the part of the map for the pieces of the other kind (Pawn vs non-Pawn). Update of the summary is more tricky, especially if we give a separate summary for the Pawn attacks; we would then need four 32-bit summaries, for PxP, PxX, XxP and XxX. The attacks by P would in principle only need 2-bit counters, as you can never be attacked by more than 2 Pawns. But it is advantageous to keep the same format as for the attacks by pieces, so that the two can be added together to get total attacks on each victim. The update would then have to determine whether an attack that needs to be altered was on a Pawn or on another piece, to know what summary word to update. But it would have to determine anyway whether the attack was on a friend (and thus really a 'protect') or on a foe, as these would have different summaries too. Determining the Pawn / non-Pawn character at the same time would be no extra work. (E.g. use int summary[8], and index it by (victim>>3) to obtain the color-indication bits and highest bit of the piece number (which disinguishes Pawns from non-Pawns) when updating.)

That leaves the move generation from the attack map itself. (Which probably is much faster than the update anyway.) If done 'mechanically' this would split all extraction loops into two (high part after low part), which involves some extra loop overhead. (Each loop will need a branch which is probably mispredicted at the end, because extraction loops have no predictable number of iterations.) The structure of the capture generator is two levels of loop nesting: an outer loop over 'value groups', containing two inner loops: the first to combine all attacker sets of the victims in the value group, and then a loop to extract the individual captures from this combination:

Code: Select all

for all value-groups in summary {
  for all victims in the value-group
    combine attackers
  for all captures in combination
    search capture
}
So on the face of it we would get two such outer loops, extracting from summaryP (attacks on Pawns) and summary_X (attacks on others), each containing 4 inner loops. But it is not as bad as that, as for value-groups in summary_X, we of course only have piece victims, and no Pawn victims. And in summary_P there is only a single value group, so that the outer loop can disappear:

Code: Select all

for all value-groups in summary_X {
  for all victims in value-group & summary_X
    combine attackers_X
    combine attackers_P
  for all captures in combination_P
    search capture // the P x X captures
  for all captures in combination_X
    search capture // the X x X captures
}

for all victims in summary_P
  combine attackers_X
  combine attackers_P
for all captures in combination_P
  search capture // P x P captures
for all captures in combination_X
  search capture // X x P captures
The splitting of the for-all-captures loops can actually be put to our advantage in a scheme for more refined move ordering, where we would want to postpone (or prune) H x L captures based on their SEE or protection of the victim. The test for this would only have to be done for captures extracted from the combinationX set, as P x X captures will never be H x L. And the X x P captures are automatically H x L, also simplifying the test. This would probably more than make up the overhead of splitting the loops.

This does not fully address the problem of the large Pawn value-group, though. With the COMB spacing of the attackers in the 32-bit words, we can still only combine the attackers sets of 4 Pawns. Now it could be that this is enough, because only 4 Pawns are attacked. (Or perhaps the opponent doesn't even have more than 4 Pawns.) But we cannot rely on that. An important point is that the LVA methodology probably suchs for the X x P captures anyway. These are all H x L, and should not be ordered by LVA but by SEE. E.g. Q x unprotected P should go before N x protected P! And when it is unprotected, the value of the piece with which you capture would be irrelevant. So we will refrain from enforcing LVA order on the X x P captures, to avoid complexity that serves no purpose.

It is a bit of a trade-off whether to split the combination loops into separate loops for getting the Pawn attackers and the other attackers on a given value group, or combine them at once. In the latter case you might waste time by adding empty attacker sets to the combination, when a victim had only Pawn attackers, or only other attackers. When separate aatcks-by-Pawn and attacks-by-other summaries are available, you could extract from those in separate loops (instead of adding them first), and would never have add an empty set to the combination. But in the case where a victim has both Pawn and non-Pawn attackers, it would now be treated in both loops, requiring an extra bit extraction and clearing. Doing separate loops has the advantage that you can do the attacks-by-Pawn combination first, and then immediately extract the captures and search them, before combining attacks-by-other and searching those, so that a beta cutoff by one of the by-Pawn captures would save you the work of combining the attacks-by-others. Al in all it seems splitting the combination loop would be better. This finally gives the following logic:

Code: Select all

summary_T = summary_PxX + summary_XxX
for all value-groups in summary_T {
  for all victims in value-group & summary_PxX
    combine attackers_P
  for all captures in combination_P
    search capture // the P x X captures
  for all victims in value-group & summary_XxX
    combine attackers_X
  for all captures in combination_X
    search capture // the X x X captures
}

for 1st 4 victims in summary_PxP
  combine attackers_P
for all captures in combination_P
  search capture // P x P captures
for 2nd 4 victims in summary_PxP
  combine attackers_P
for all captures in combination_P
  search capture // P x P captures

for 1st 4 victims in summary_XxP
  combine attackers_X
for all captures in combinationX
  search capture // X x P captures
for 2nd 4 victims in summaryXxP
  combine attackers_X
for all captures in combinationX
  search capture // X x P captures
So the bottom line is: we will use 32-bit integers for the attack map. Each piece will have four elements in the attack map associated with it, each containing flags for 8 attackers, in a COMB pattern: white Pawns, black Pawns, white other and black other. There will be 16 summaries, each consisting of eight packed 4-bit counters indicating the total number of attacks on eight victims of the same kind (P vs non-P) and color. That we need 16 is because they distinguis by color of the attacker and the victim, as well as the P/non-P character of the attacker and the victim. For move generation we would only need the four summaries for attackers of the sid-to-move color, and victims of his opponent. The equal-colored summaries (protections) are for assisting in the incremental update, to make it easy for a capturer to inherit the old protectors of his victim as new attackers, and vice versa. They could of course also be useful when determining whether a victim is protected or not.
Mike Sherwin
Posts: 860
Joined: Fri Aug 21, 2020 1:25 am
Location: Planet Earth, Sol system
Full name: Michael J Sherwin

Re: The mailbox trials

Post by Mike Sherwin »

:? That left me face down in the dirt.
User avatar
hgm
Posts: 27788
Joined: Fri Mar 10, 2006 10:06 am
Location: Amsterdam
Full name: H G Muller

Re: The mailbox trials

Post by hgm »

Well, perhaps it helps when I give some actual code. For creating an attack map from scratch one could use the following routine. Which is very much like the CaptGen3() I had in the previous mailbox version, except that it generates moves for only one piece, which is known to be present. So there is no loop over the non-captured pieces in the piece list. And it does not flush the captures it geenrates to a move list, but sets the corresponding bits in the attack map:

Code: Select all

//
// Piece encoding  EMPTY          16 = WHITE      32 = BLACK      48 = COLOR (edge guard)
//                 |               |               |               |
//                 0...............PPPPPPPPNNBBRRQKppppppppnnbbrrqk#
//                  (1-15 unused)  (16- 23)(24- 31)(32- 39)(40- 47)
//
// Piece data that is never consulted for empty squares have 32 elements, and are indexed by piece-16.
// If we examine the binary representation of piece numbers, the bits have the following function:
//
//                  MSB         LSB        nnn = number (implying type for non-Pawns)
//                  . . B W P n n n        P = Pawn if 0, non-Pawn if 1
//                                         W, B: White or Black
//
// Note that the WP bits identify the 'kind' of piece if we know the code represents a piece (and B = !W)
// If the code can also be an edge guard (B = W = 1) or empty square (all 0), we need BWP to identify that


int victim2unit[] = { // unit in summary counters, per piece (extract in MVV order!)
  1<<28, 1<<24, 1<<20, 1<<16, 1<<12, 1<<8, 1<<4, 1, // PPPPPPPP
  1<<28, 1<<24, 1<<20, 1<<16, 1<<12, 1<<8, 1<<4, 1, // NNBBRRQK
  1<<28, 1<<24, 1<<20, 1<<16, 1<<12, 1<<8, 1<<4, 1, // pppppppp
  1<<28, 1<<24, 1<<20, 1<<16, 1<<12, 1<<8, 1<<4, 1, // nnbbrrqk
};

int attacker2bit[] = { // flags used to indicate attacker, per piece (extract in LVA order!)
 1, 1<<4, 1<<8, 1<<12, 1<<16, 1<<20, 1<<24, 1<<28, // PPPPPPPP
 1, 1<<4, 1<<8, 1<<12, 1<<16, 1<<20, 1<<24, 1<<28, // NNBBRRQK
 1, 1<<4, 1<<8, 1<<12, 1<<16, 1<<20, 1<<24, 1<<28, // pppppppp
 1, 1<<4, 1<<8, 1<<12, 1<<16, 1<<20, 1<<24, 1<<28, // nnbbrrqk
};

/*
    The attack map logically is a 32x32 array of bit flags, indicating what attacks what.
    It is stored as 128 groups of 8 bits, containing 8 different attackers of the same victim:
                           attackers
                     white           black
                 pawns   other   pawns   other
                PPPPPPPPNNBBRRQKppppppppnnbbrrqk
               P................................
             p P................................
             a P................................
             w P................................
             n P................................
          w  s P********........................   * P protectors of white P #6
          h    P................................
          i    P................................
          t    N................................
          e  o N................................
             t B................................
        v    h B........................@@@@@@@@   @ piece attackers of white B #2
        i    e R................................
        c    r R................................
        t      Q................................
        i      K................................
        m      p................................
        s    p p........########................   # piece attackers of black P #2
             a p................................
             w p................................
             n p................................
          b  s p................................
          l    p................................
          a    p................................
          c    n++++++++........................   + summary of white P attacks on black pieces
          m  o n++++++++........................     each row is summarized by one 4-bit counter
             t b++++++++........................
             h b++++++++........................
             e r++++++++........................
             r r++++++++........................
               q++++++++........................
               k++++++++........................

    Its elements ('attackers sets') have the format:

                         MSB                          LSB  (. = unused bit, always 0)
                         ...K...Q...R...R...B...B...N...N  8 piece attackers
                     or  ...P...P...P...P...P...P...P...P  8 Pawn attackers

    Note the assignment of pieces to bits is such that they extract in order of increasing value.
    This is done so that attackers sets on different victims can be easily merged by shift and add:

                         ...K...Q...R...R...B...B...N...N  8 piece attackers of Rook #1
                        ...K...Q...R...R...B...B...N...N   8 piece attackers of Rook #2
                        _________________________________ +
                         ..KK..QQ..RR..RR..BB..BB..NN..NN  All captures of Rooks

    Such merging preserves the LVA order of the extraction, all NxR and BxR before any RxR, etc.

    Each of the 128 attackers sets is 'summarized' as the number of attacks in it, by a 4-bit counter.
    Eight such counters are packed into one 32-bit integer.
    Counters that summarize attacks by pieces of the same kind on pieces of the same kind go together,
    where 'kind' is one of white Pawn, white non-Pawn, black Pawn, black non-Pawn.
    So the area marked by + in the diagram above is summarize in one summary word,
    which contains one 4-bit counter for each row (so that the counter corresponds to a victim).
    The format of the summary elements is:

                          MSB                          LSB
                          (N1)(N2)(B1)(B2)(R1)(R2)( Q)( K)

    The assignment of pieces to counters is such that they extract in order of decreasing value.

*/

int attackers[4*32]; // attackers set per victim, for 4 different kinds of attackers (bP, bX, wP, wX)
int summary[4*4];    // (packed) number of attacks on victims of a certain kind, per attacker & victim kind

void AddMoves(int piece, int sqr, int add)
{
  int offs = (piece & 030); // offsets: 0 = black P, 8 = black other, 16 = white P, 24 = white other
  int *sum = summary + offs;           // part of summary[] to use for this kind of attacker
  int *atts = attackers + 4*offs - 16; // part of attackers[] to use for this kind of attacker
  int attBit = attacker2bit[piece-16]*add;
  if(code[piece] & 0200) { // piece is a slider
    int d, i = firstSlide[slideOffs[piece-16] + sqr]; // list of directions for this square
    while((d = slides[i++]) >= 0) {                   // loop over directions
      int to = neighbor[from].d[d];                   // neighbor in this direction
      int victim = board[to];                         // occupant of the neighbor square
      if(victim < COLOR) {                            // rejects edge guards (cannot be empty!)
        int victimKind = victim >> 3;                 // 2 = wP, 3 = wX, 6 = bP, 7 = bX
        sum[victimKind] += victim2unit[victim-16];    // inc/dec nr of attacks on this victim
        atts[victim] += attBit;                       // set or clear flag for this attacker
      }
    }
  } else { // piece is a leaper
    int v, i = firstLeap[slideOffs[piece-16] + sqr]; // list of steps it can make from this square
    while((v = steps[i++])) {                        // loop through steps
      int to = from + v;                             // target square
      int victim = board[to];                        // occupant of the target square
      if(victim + 16 & 32) {                         // this rejects empty (0) / edge guard (48)
        int victimKind = victim >> 3 & 3;            // 2 = wP, 3 = wX, 0 = bP, 1 = bX
        sum[victimKind] += victim2unit[victim-16];   // inc/dec nr of attacks on this victim in summary[]
        atts[victim] += attBit;                      // set or clear flag for this attacker in attackers[]
      }
    }
  }
}
The tricky thing here is to determine where in the array attackers[] the bit flag we have to set resides, and where the 4-bit counter in the summary[] array resides, both in terms of the index and the location within that word. For these locations we have the tables attacker2bit[] and victim2unit[]. The array element in which this goes is determined by the victim and the 'kind' of attacker for the attack map, and the kind of attacker and kind of victim for the summaries.

As mentioned, this routine could build an attack map from scratch, by calling it for all pieces that are present. But we want to update the map incrementally, and then we only have to call it for the piece that moved. But we would have to be able to remove its moves from the old location. For that the routine has a parameter 'add', which can be set to -1, so that the modifications are subtracted rather than added.

As an example of how the map will be used, there is the following move-generation loop:

Code: Select all

unsigned int valueGroup[] = { // masks for selecting the packed counters, per summary bit
  0xF, 0xF, 0xF, 0xF, 0xF,                0xF0, 0xF0, 0xF0, 0xF0, // K & Q counter
  0xFF00, 0xFF00, 0xFF00, 0xFF00, 0xFF00, 0xFF00, 0xFF00, 0xFF00, // R counters
  0xFFFF0000, 0xFFFF0000, 0xFFFF0000, 0xFFFF0000, 0xFFFF0000, 0xFFFF0000, 0xFFFF0000, 0xFFFF0000, // B
  0xFFFF0000, 0xFFFF0000, 0xFFFF0000, 0xFFFF0000, 0xFFFF0000, 0xFFFF0000, 0xFFFF0000, 0xFFFF0000, // N
};

int bit2unit[] {
  1, 1, 1, 1, 1<<4, 1<<4, 1<<4, 1<<4, 1<<8, 1<<8, 1<<8, 1<<8, 1<<12, 1<<12, 1<<12, 1<<12,
  1<<16, 1<<16, 1<<16, 1<<16,
};

// capture generation for white

int victims[8];    // victim stack
int victimSP = 0;  // victim stack pointer
int captures;

// first do all captures of non-Pawn victims (as MVV ordering prescribes)
int todo = summary[16+1] + summary[24+1];   // attacks by white P (16) and X (24) on black X (1)
while(todo) {                               // loop over value groups ({K}, {Q}, {R,R}, {B,B,N,N})
  int bit = BSF(todo);                      // locate lowest non-zero counter
  int groupMask = valueGroup[bit];          // bits of counters for the group that counter belongs to
  // first do the P x non-P captures
  int group = groupMask & summary[16+1];    // isolate counters for Pawn attacks on this group
  if(group) {                               // this could be empty (if there are only non-Pawn attackers)
    victimSP = 0;                           // clear victim stack
    captures = 0;                           // to accumulate captures on this group
    do {                                    // loop over victims in this group attacked by Pawns
      int bit = BSF(group) & ~3;            // locate lowest non-zero counter
      int counterNr = bit >> 2;             // divide by 4 (rounding down), as counters have 4 bits each
      int victim = BLACK + 15 - counterNr;  // we are working on black non-Pawns, (MVV order!)
      captures <<= 1;                       // shift flags to make room for new
      captures += attackers[64 + victim-16];// 64 = white P attackers
      victimStack[victimSP++] = victim;     // remember victim
      group &= ~(15 << bit);                // this counter is now treated, so clear
    } while(group);
    do {                                    // loop over the constructed set of captures
      int bit = BSF(captures);              // extract next P x X capture (MVV/LVA order)
      info.victim = victimStack[bit & 3];   // get victim
      info.piece = WHITE + (bit >> 2);      // calculate white Pawn number (16-23)
      if(SearchMove(&info)) goto cutoff;    // search move and minimax score (in info struct)
      captures &= captures - 1;               
    } while(captures);
  }
  // then do the non-P x non-P captures
  ... // very similar code as above
  
  todo &= ~groupMask;                     // all victims of this value are now treated, so clear
}
Mike Sherwin
Posts: 860
Joined: Fri Aug 21, 2020 1:25 am
Location: Planet Earth, Sol system
Full name: Michael J Sherwin

Re: The mailbox trials

Post by Mike Sherwin »

I'm not sure that pure MVV-LVA is always best. What if in WQxBQ the BQ is undefended or if in WRxBQ the BQ is defended? Considering only Qsearch which is 80+% of all searching and on average produces short move list I prefer to overweight the victim, (value_victim << 4) - value_attacker.

HGM, Does your code account for in Qsearch whether or not the victim is defended?
User avatar
hgm
Posts: 27788
Joined: Fri Mar 10, 2006 10:06 am
Location: Amsterdam
Full name: H G Muller

Re: The mailbox trials

Post by hgm »

No, currently it doesn't. Because it doesn't know whether something is protected or not. For that it would have to generate opponent moves. Either all, or selectively to pieces threatened by HxL attacks. It could then do a SEE or BLIND analysis.

In the version I am aiming for this would be much easier. Because an attack map would be available. I would just have to look at the protection counter of a victim to see how many protectors it has. That would not include X-ray protectors, but 0 normal protectors implies no X-ray protectors either.

I guess the best approach would be to handle this in the stage where the captures get searched, by splitting the loop that extracts them from the combined attackers set for the current value group. A group-dependent bit mask can be used to split the attackers set into HxL and other captures. The other captures can be extracted and search unconditionally first. The following loop would then do the HxL, and would contain logic for testing protection.

Code: Select all

int safeCapts = captures & hxlMask[valueGroupNr]; // mask away all more valuable attackers
captures -= safeCapts; // remove those from the initial set
while(safeCapts) {
  int bit = BSF(saveCapts);
  info.victim = victimStack[bit & 3];
  info.piece = WHITE + 8 + (bit >> 2);
  if(SearchMove(&info)) goto cutoff;
  safeCapst &= safeCapts - 1;
}
int protectors = summary[BLACKPIECES][BLACKPIECES]; // black non-P x black non-P counters
protectors += summary[BLACKPAWNS][BLACKPIECES];   // add the black P x black non-P counters to get total
while(captures) { // second move loop for remaining HxL captures
  int bit = BSF(captures);
  captures &= captures - 1;
  info.victim = victimStack[bit & 3];
  if(protectors & 15*victim2unit[info.victim]) { // victim is protected
    captures &= 0xEEEEEEEE * bit; // mask away all higher attackers of this victim
  } else { // unprotected: capture as usual
    info.piece = WHITE + 8 + (bit >> 2);
    if(SearchMove(&info)) goto cutoff;
  }
}
This doesn't solve the Q x protected Q vs R x unprotected Q problem. But your lower weighting of the victim doesn't do that either, as it doesn't factor in protection. And I am not sure your premise that R x unprotected Q should go first is actually true. The rationale behind MVV is that after the QxQ, you would get to play the RxQ anyway if he recaptures. And if he doesn't, the protection might as well not have existed. OTOH, if you start with R x unprotected Q, he would reply with Q x Q, and perhaps your own Q was also unprotected (in which case you gained nothing).

This completely prunes all H x protected L captures, which might be overdoing it. But even in this design there is a conventional move list, for handling the non-captures. So instead of masking away the higher attackers you could just flush the suspect capture to the normal move list, for later search (after the lower-valued victims have been tried). Since the number of Pawn protectors is available separately from the number of other protectors, you could even make this dependent on the presence of Pawn protectors (which would make it a bad capture for sure).