How to reduce the "bits" used in a magic number ca
Moderators: hgm, Rebel, chrisw

 Posts: 316
 Joined: Wed Apr 12, 2006 10:47 pm
Re: How to reduce the "bits" used in a magic numbe
Hi, isn't there a more mathematical way of finding possible magic numbers, using random numbers doesn't seem the best way of doing it when there are too many possibilities for a computer to test in a small time.
Colin

 Posts: 287
 Joined: Sat Mar 11, 2006 3:19 am
 Location: Atlanta, GA
Re: How to reduce the "bits" used in a magic numbe
Yes, there is a deterministic way of generating magics as I outlined here. I have used this method to generate both optimal magics (both "indexoptimal" and "bitoptimal") for some bishop squares and have used it to prove that no minimalbit magic key (6bits because there are 49 possible moves) exists for square 0 for rooks. However, the way I've been doing it right now, I use a very adhoc order of guessing the bits in the magic. If the following problem can be solved then the speed of this method of magic generation can be improved by many orders of magnitude and I believe that it may even be able to deterministically generate optimal magics for both rooks and bishops for all squares:cms271828 wrote:Hi, isn't there a more mathematical way of finding possible magic numbers, using random numbers doesn't seem the best way of doing it when there are too many possibilities for a computer to test in a small time.
Given the input keys, the allowable collision groups, and the bits in the magic guessed so far, what is the best bit to guess next and what value should be guessed for it first (1 or 0) such that the probability of a fast collision or a cutoff can be maximized?
It is a rather difficult problem and I haven't had enough time to sit down and think over it yet, but I'm sure if I or someone else figure it out, it should be possible to generate optimal magics for both bishops and rooks.

 Posts: 2251
 Joined: Wed Mar 08, 2006 8:47 pm
 Location: Hattingen, Germany
Re: How to reduce the "bits" used in a magic numbe
So far it seems you still need trial and error and the deterministic way is about a first guess for most squares with up to four rays with length > 1. What do you mean with "indexoptimal" and "bitoptimal"?Pradu wrote:Yes, there is a deterministic way of generating magics as I outlined here. I have used this method to generate both optimal magics (both "indexoptimal" and "bitoptimal") for some bishop squares and have used it to prove that no minimalbit magic key (6bits because there are 49 possible moves) exists for square 0 for rooks. However, the way I've been doing it right now, I use a very adhoc order of guessing the bits in the magic. If the following problem can be solved then the speed of this method of magic generation can be improved by many orders of magnitude and I believe that it may even be able to deterministically generate optimal magics for both rooks and bishops for all squares:cms271828 wrote:Hi, isn't there a more mathematical way of finding possible magic numbers, using random numbers doesn't seem the best way of doing it when there are too many possibilities for a computer to test in a small time.
Given the input keys, the allowable collision groups, and the bits in the magic guessed so far, what is the best bit to guess next and what value should be guessed for it first (1 or 0) such that the probability of a fast collision or a cutoff can be maximized?
It is a rather difficult problem and I haven't had enough time to sit down and think over it yet, but I'm sure if I or someone else figure it out, it should be possible to generate optimal magics for both bishops and rooks.
In general it is about mapping from N scattered relevant occupied bits to N (or N1) consecutive upper bits, taking as much advantage of constructive collisions as possible, where different occupancies have common attack sets due to redundant outer occupied bits.
For instance a oneray pattern, say bishop on a1. It is simple to find a kindergarten like fillfactor for a 1:1 mapping of all six relevant occupied bits  without any carries and collisions:
Code: Select all
idx6bitBishopA1 = ((occ & 0x0040201008040200) * 0x0002020202020200) >> (646)
Code: Select all
idx5bitBishopA1 = ((occ & 0x0040201008040200) * 0xffedf9fd7cfcffff) >> (645)
How can we prove (for this bishop square a1), what the most dense index range is?
Code: Select all
idx4bitBishopA1 = ((occ & 0x0040201008040200) * magic4) >> (644)
idx3bitBishopA1 = ((occ & 0x0040201008040200) * magic3) >> (643)

 Posts: 287
 Joined: Sat Mar 11, 2006 3:19 am
 Location: Atlanta, GA
Re: How to reduce the "bits" used in a magic numbe
Yes you still need trial and error (that's how DeBruijns are generated too) but the important thing is that technique is deterministic. "Bitoptimal" refers to a magic that reduces the number of bits in the index. "Indexoptimal" refers to a magic that reduces the largest index that the function hashes to. Of course an indexoptimal magic is a bitoptimal magic.Gerd Isenberg wrote:So far it seems you still need trial and error and the deterministic way is about a first guess for most squares with up to four rays with length > 1. What do you mean with "indexoptimal" and "bitoptimal"?
Simply by running through the entire searchtree of viable bitoptimal magics (the same way how we did it to find the largest 64bit DeBruijn magic, just go through them all and find the biggest one). This is indeed possible as I've already shown for rook A1 for 6bits even with an adhoc bit guessing scheme which took an over night run. If running through the entire search tree does not produce a viable magic, then it is proven that such a magic does not exist. For producing indexoptimal magics you may be able to use more cutoffs if you can determine up front that your magic so far will hash to an index larger than or equal to the best magic found so far. I'll try to produce an indexoptimal magic for bishops for A1 as an examplehowever I won't be able to work on it for a few days as I'll be out of town...How can we prove (for this bishop square a1), what the most dense index range is?
You could probably find the indexoptimal magics for bishops for the upper squares even without trialerror treesearching and cutoffs. For example you could try the "easiest" bishop square D8 because it has so many upperredundant bits and you can do this simply by trying through all the numbers from 0>2^n1 where n=64(number of upperredundant bits). Similarly, you can run through the searchtree of viable magics for say Bishop on A1 and you'll pick the one that gives the smallest index.

 Posts: 317
 Joined: Mon Jun 26, 2006 9:44 am
Re: How to reduce the "bits" used in a magic numbe
I don't know about generating a complete set of nbit DeBruijn numbers for a given n, but generating at least 1 for a given n can be done deterministically and efficiently.

 Posts: 287
 Joined: Sat Mar 11, 2006 3:19 am
 Location: Atlanta, GA
Re: How to reduce the "bits" used in a magic numbe
It can be generated in a maximum of 30 minutes. Gerd has a faster generator for this that employs early cutoffs and IIRC it can generate them all in 1 minute.rjgibert wrote:I don't know about generating a complete set of nbit DeBruijn numbers for a given n, but generating at least 1 for a given n can be done deterministically and efficiently.

 Posts: 287
 Joined: Sat Mar 11, 2006 3:19 am
 Location: Atlanta, GA
Re: How to reduce the "bits" used in a magic numbe
I think I came up with a good bit guessing scheme over the weekend and also managed to come up with a better collision detection scheme:
New collision detection scheme
Let keys k0, k1, ... kn collide onto the same index given the guessed bits so far g and the unguessed bits so far u. How can it be determined if no combination of the unguessed bits can prevent the collision?
Let d = ki^kj for ki and kj being an unwanted collision. If d*u for some combination of the unguessed bits u can affect the index bits, then the collision can be prevented, otherwise it can't be prevented. The routine that will determine if no possible value of u can prevent the collision will be as follows:
New bit guessing scheme
I decided that determining the best order of guessing upfront would be too difficult. So I came up with a heuristic that uses a 1 ply search to order the bits. And here's how it works:
When you get through this loop, guessing a 0 of a 1 for any bit will not cause a cutoff. When trying to check for cutoffs you could at the same time count the number of unwanted collisions that occur. The next bit and value you would want to guess is the one that produces the most unwanted collisions. If two or more bits produce the same number of unwanted collisions, then pick the bit that is less significant. Less significant bits are more important than more significant bits because it is less likely that they are redundant for a certain subset of the input keys than the more significant bits.
I'll try to implement the above into my existing generator when I get the time in the next few days.
New collision detection scheme
Let keys k0, k1, ... kn collide onto the same index given the guessed bits so far g and the unguessed bits so far u. How can it be determined if no combination of the unguessed bits can prevent the collision?
Let d = ki^kj for ki and kj being an unwanted collision. If d*u for some combination of the unguessed bits u can affect the index bits, then the collision can be prevented, otherwise it can't be prevented. The routine that will determine if no possible value of u can prevent the collision will be as follows:
Code: Select all
/**
* This routine returns a bitboard of all bits that aren't upper redundant.
* key  input key
* s  number of bits in the index
*/
U64 notRedundant(const U64 key)
{
return (U64FULL>>FirstOne(key));
}
/**
* This routine returns true if collision between two keys is not preventable.
* k1  first key
* k2  second key
* unguessed  the unguessed bits
* s  the number of bits in the index
*/
bool notPreventable(const U64 k1, const U64 k2, U64 unguessed, const int s)
{
U64 d = k1^k2; /* only the difference in the keys matter */
/* max is the smallest number that gets into the index bits */
U64 max = (C64(1)<<(64s));
unguessed &= notRedundant(d);
while(unguessed)
{
U64 b_i = LSB(unguessed); /* For each unguessed bit */
U64 map = d*b_i; /* Compute the index mapping */
if(map>=max) return false; /* collision may be preventable */
max=map;
unguessed^=b_i;
}
return true; /* collision is not preventable */
}
I decided that determining the best order of guessing upfront would be too difficult. So I came up with a heuristic that uses a 1 ply search to order the bits. And here's how it works:
Code: Select all
Do
For all unguessed bits
Try both 0 and 1 for a bit.
If both 0 and 1 causes a cutoff, then this node cuts off.
If 0 causes a cutoff, then the value of this bit is determined to be 1.
If 1 causes a cutoff, then the value of this bit is determined to be 0.
End For
While(at least one bit caused a cutoff)
I'll try to implement the above into my existing generator when I get the time in the next few days.

 Posts: 67
 Joined: Mon Aug 06, 2007 4:42 pm
 Location: London, England
Re: How to reduce the "bits" used in a magic numbe
Hi
I'm not at my home computer right now so I can't try this out, but is it possible that when we calculate the attackboard we can do away with the shift lookup table by incorporating the shift into the magic number?
So instead of
do
This looks favourable in my 32bit code.
Grant
I'm not at my home computer right now so I can't try this out, but is it possible that when we calculate the attackboard we can do away with the shift lookup table by incorporating the shift into the magic number?
So instead of
Code: Select all
index = (int)((occupied * rookMagic[square]) >> rookShift[square]);
Code: Select all
index = (int) ((occupied * rookMagic[square]) >> (rookMagic[square] >> 58));
Grant

 Posts: 838
 Joined: Thu Jul 05, 2007 5:03 pm
 Location: British Columbia, Canada
Re: How to reduce the "bits" used in a magic numbe
Unlikely. The magic number causes the desired bits to be collected in the upper bits of the result (along with some stray bits elsewhere). The shift is then needed to isolate them. It is analogous to computing a fixedpoint multiplication result and then rounding it to an integer. Unless you could think of some way to only compute the upper bits of the result (e.g. if using 32bit muls to implement the 64bit operation, or something).grant wrote:I'm not at my home computer right now so I can't try this out, but is it possible that when we calculate the attackboard we can do away with the shift lookup table by incorporating the shift into the magic number?

 Posts: 287
 Joined: Sat Mar 11, 2006 3:19 am
 Location: Atlanta, GA
Re: How to reduce the "bits" used in a magic numbe
Yes it is possible.grant wrote:Hi
I'm not at my home computer right now so I can't try this out, but is it possible that when we calculate the attackboard we can do away with the shift lookup table by incorporating the shift into the magic number?
So instead ofdoCode: Select all
index = (int)((occupied * rookMagic[square]) >> rookShift[square]);
This looks favourable in my 32bit code.Code: Select all
index = (int) ((occupied * rookMagic[square]) >> (rookMagic[square] >> 58));
Grant