I will most certainly update it as soon as I've finished the new implementation and will also try to make it more rigorous and easier to read. I will also post the sources to the magic-generator as well (I didn't post the sources for my old one because it was really hacked up and poorly documented).
How the above improvements fit in the existing algorithm is as follows:
- Generate input keys (occupancies) and give each input key a group number so that two input keys with the same group number have the same set of moves.
- Pick the number of bits in the index that you'd like to try to generate a magic for.
- A variable magic can be represented by two 64-bit integers. All unguessed bits are '1' in u and all guessed bits are '0' in u. All unguessed bits are '0' in g and all guessed bits are either '0' or '1' in g depending on the guessed value.
- First, all upper-redundant bits in the variable magic are set to 0 by u&=(U64FULL>>FirstOne(preMask[sq]))
- Lets consider using a guessing order where you always guess 0 first and 1 next and you guess bits from bit 0 to bit 63 and lets start a recursive bit-guessing search using this guessing order:
- First you perform a 1-ply search of guessing bits to see if some bits can be determined upfront. When you find out that you can determine the value of a bit you set its value, you can repeat the 1-ply search until no bits can be determined. This is where the new improvements (the one using difference keys that tries to determine if some bad collisions are unpreventable regardless of what bits you guess) come in. After you guess a bit you can check to see if you can determine upfront if the bit guessed will lead to an unpreventable bad collision. If both values (0 and 1) will cause an unpreventable collision you can return from this node (no value of this bit can help prevent a bad collision). If only 0 causes an unpreventable collision, then the value of the bit must be guessed as 1. If only 1 causes an unpreventable collision, then the value of the bit must be guessed as 0.
- After the 1-ply search you can guess the next bit which is determined by your guessing order.
The recursive search will eventually guess all the bits without unwanted collisions and it will produce a magic number. Then you can return from the leaf-node and continue the search to find another magic number. And in this process you can go through them all and pick the one that produces the smallest index for the given number of bits. If there's no magic number for the number of bits that you've chosen then the recursive search will never get to a leaf node and so it'll never produce a magic number.
Looking for all possible magics for a given pattern has an unrestricted search tree of 2^64 = 1.8*10^19 nodes. This is equivalent in size to the search tree of the game of checkers. Now of course, checkers has been solved, by basically three search improvements: alpha-beta cutoffs in combination with hashtables and endgame databases.
Since the search tree for magics is already ordered, there are no transpositions and hashtables are of no use. The method to identify unpreventable collissions by difference keys is the equivalent of an alpha-beta cutoff. It should also possible to combine this with the equivalent of endgame databases. Analogously to pre-computing the outcome of all endgames with a few pieces on the board, one could pre-compute the unpreventability of collissions for all magics with a few bits set to 1.
E.g. a 2-bit database is a table of all 64*63/2 pairs of bits that are to be guessed in the magic. Each pair can take on 2^2 values. In all, there are about 8K entries in the table. Each entry states whether the values of a pair of bits in a magic lead to an unpreventable collission. Similarly, a 3-bit database has 64*63*62*/6 triplets of bits, with 2^3 values for each triplet. All in all about 33K values. These tables will have to be computed once for every pattern before searching for a magic.
Step 5.a) of Pradu's algorithm can then be extended by first running over the magic guessed so far to see whether there is any 2-bit or 3-bit pattern present in the set of guessed bits that will always lead to a collission. If so, then terminate this node. If not, proceed by guessing the next bit, see if this in isolation will lead to an unpreventable collission etc.
If the percentage of disallowed pairs or triplets of bits is large, and the cost of searching for such patterns is low, then a considerable speedup might be gained.