Computing hash "on the fly"

Discussion of chess software programming and technical issues.

Moderators: hgm, Rebel, chrisw

User avatar
stegemma
Posts: 859
Joined: Mon Aug 10, 2009 10:05 pm
Location: Italy
Full name: Stefano Gemma

Computing hash "on the fly"

Post by stegemma »

In Satana I was computing hash value move by move, removing previous "from-to" squares hash value and xor the new one. Now I'm testing the "on the fly" computing of hash, not in the ExecuteMove but directly using the bitboards. It is still to refine (I have some false-positive hash hits) but it seems faster than previous approach: about 6400 Knps vs 5900 Knps, with about 20% of cache hits for both methods:

Code: Select all

uint64_t clsBoard::GetHash()
{
#if FAST_TT
	uint64_t boK = (boAllPieces ^ (boAllPieces >> 31) + boPawns);
	boHash = boK
		^ (boKings * ~boK)
		^ (boQueens * ~boK)
		^ (boRooks * ~boK)
		^ (boKnights * ~boK)
		^ (boPawns * ~boK)
		^ (boAllPieces / 17)
		;
	return boHash ^ boAllPieces ^ (boEnpPawn - boCastlings) ^ zobcol[color];
#else
	return boHash ^ boAllPieces ^ (boEnpPawn - boCastlings) ^ zobcol[color];
#endif
}
Anybody use this kind of hash computing?

PS: the "/17" will be removed soon...
Author of Drago, Raffaela, Freccia, Satana, Sabrina.
http://www.linformatica.com
Joost Buijs
Posts: 1563
Joined: Thu Jul 16, 2009 10:47 am
Location: Almere, The Netherlands

Re: Computing hash "on the fly"

Post by Joost Buijs »

stegemma wrote:In Satana I was computing hash value move by move, removing previous "from-to" squares hash value and xor the new one. Now I'm testing the "on the fly" computing of hash, not in the ExecuteMove but directly using the bitboards. It is still to refine (I have some false-positive hash hits) but it seems faster than previous approach: about 6400 Knps vs 5900 Knps, with about 20% of cache hits for both methods:

Code: Select all

uint64_t clsBoard::GetHash()
{
#if FAST_TT
	uint64_t boK = (boAllPieces ^ (boAllPieces >> 31) + boPawns);
	boHash = boK
		^ (boKings * ~boK)
		^ (boQueens * ~boK)
		^ (boRooks * ~boK)
		^ (boKnights * ~boK)
		^ (boPawns * ~boK)
		^ (boAllPieces / 17)
		;
	return boHash ^ boAllPieces ^ (boEnpPawn - boCastlings) ^ zobcol[color];
#else
	return boHash ^ boAllPieces ^ (boEnpPawn - boCastlings) ^ zobcol[color];
#endif
}
It is essential that you encode both piece-type and piece-location in your hash value, in your example this doesn't seem to happen.
For instance when you exchange location of the 2 knights with the location of the 2 rooks you will get the same hash value etc.

NPS doesn't say much, when the search tree is shaped a little bit different because of the altered hash calculation it may generate more quiescence nodes which are usually faster compared to full nodes.
User avatar
stegemma
Posts: 859
Joined: Mon Aug 10, 2009 10:05 pm
Location: Italy
Full name: Stefano Gemma

Re: Computing hash "on the fly"

Post by stegemma »

I have already changed the formula, to avoid this problem. It gives good hashing from the starting position and when all pieces/pawns are on the chessboard but poor hashing when you have few pieces. I'm experimenting various options.

(I've quiescence disabled, so the nps is a good indicator, in my engine)
Author of Drago, Raffaela, Freccia, Satana, Sabrina.
http://www.linformatica.com
AlvaroBegue
Posts: 931
Joined: Tue Mar 09, 2010 3:46 pm
Location: New York
Full name: Álvaro Begué (RuyDos)

Re: Computing hash "on the fly"

Post by AlvaroBegue »

That particular code is full of problems. For instance, think of what would happen if ~bOK happens to end in a whole bunch of 0s. Then the upper bits of boKings, boQueens, etc. are ignored in your computation.

Computing hash keys on the fly is doable, but you need to be more careful than that.
bob
Posts: 20943
Joined: Mon Feb 27, 2006 7:30 pm
Location: Birmingham, AL

Re: Computing hash "on the fly"

Post by bob »

stegemma wrote:I have already changed the formula, to avoid this problem. It gives good hashing from the starting position and when all pieces/pawns are on the chessboard but poor hashing when you have few pieces. I'm experimenting various options.

(I've quiescence disabled, so the nps is a good indicator, in my engine)
You are trying to re-invent a wheel that has never worked. :) Zobrist has been THE way to produce a hash signature since the 70's. It is critical that each piece have an influence on the final hash signature, and that you can't interchange pieces and get the same signature...
User avatar
stegemma
Posts: 859
Joined: Mon Aug 10, 2009 10:05 pm
Location: Italy
Full name: Stefano Gemma

Re: Computing hash "on the fly"

Post by stegemma »

bob wrote:
stegemma wrote:I have already changed the formula, to avoid this problem. It gives good hashing from the starting position and when all pieces/pawns are on the chessboard but poor hashing when you have few pieces. I'm experimenting various options.

(I've quiescence disabled, so the nps is a good indicator, in my engine)
You are trying to re-invent a wheel that has never worked. :) Zobrist has been THE way to produce a hash signature since the 70's. It is critical that each piece have an influence on the final hash signature, and that you can't interchange pieces and get the same signature...
Ok, I already have Zobrist key working but I'm not satisfied in the speed of my implementation, that's why I'm experimenting. I know that I'll lose a few hours but this useless work will let me more comfortable with TT, that works very bad in Satana.
Author of Drago, Raffaela, Freccia, Satana, Sabrina.
http://www.linformatica.com
AlvaroBegue
Posts: 931
Joined: Tue Mar 09, 2010 3:46 pm
Location: New York
Full name: Álvaro Begué (RuyDos)

Re: Computing hash "on the fly"

Post by AlvaroBegue »

stegemma wrote:
bob wrote:
stegemma wrote:I have already changed the formula, to avoid this problem. It gives good hashing from the starting position and when all pieces/pawns are on the chessboard but poor hashing when you have few pieces. I'm experimenting various options.

(I've quiescence disabled, so the nps is a good indicator, in my engine)
You are trying to re-invent a wheel that has never worked. :) Zobrist has been THE way to produce a hash signature since the 70's. It is critical that each piece have an influence on the final hash signature, and that you can't interchange pieces and get the same signature...
Ok, I already have Zobrist key working but I'm not satisfied in the speed of my implementation, that's why I'm experimenting. I know that I'll lose a few hours but this useless work will let me more comfortable with TT, that works very bad in Satana.
Can you post some code from your implementation? Updating the Zobrist key incrementally as the board changes should be very fast.

My low-level code that modifies the board looks something like this:

Code: Select all

void Board::place_piece(int square, int piece) {
  a[square] = piece;
  bb&#91;piece&#93; |= 1ull << square;
  zobrist_key ^= zobrist_table&#91;piece&#93;&#91;square&#93;;
&#125;

void Board&#58;&#58;remove_piece&#40;int square, int piece&#41; &#123;
  a&#91;square&#93; = Empty;
  bb&#91;piece&#93; &= ~&#40;1ull << square&#41;;
  zobrist_key ^= zobrist_table&#91;piece&#93;&#91;square&#93;;
&#125;

The the final hash key is computed by this code:

Code: Select all

u64 Board&#58;&#58;get_hash_key&#40;) const &#123;
  return zobrist_key
    ^ en_passant_zobrist_table&#91;en_passant&#93;
    ^ castling_availability_zobrist_table&#91;castling_availability&#93;
    ^ side_to_move_zobrist_table&#91;side_to_move&#93;);
&#125;
syzygy
Posts: 5557
Joined: Tue Feb 28, 2012 11:56 pm

Re: Computing hash "on the fly"

Post by syzygy »

stegemma wrote:
bob wrote:
stegemma wrote:I have already changed the formula, to avoid this problem. It gives good hashing from the starting position and when all pieces/pawns are on the chessboard but poor hashing when you have few pieces. I'm experimenting various options.

(I've quiescence disabled, so the nps is a good indicator, in my engine)
You are trying to re-invent a wheel that has never worked. :) Zobrist has been THE way to produce a hash signature since the 70's. It is critical that each piece have an influence on the final hash signature, and that you can't interchange pieces and get the same signature...
Ok, I already have Zobrist key working but I'm not satisfied in the speed of my implementation, that's why I'm experimenting. I know that I'll lose a few hours but this useless work will let me more comfortable with TT, that works very bad in Satana.
If it works very bad, then your problem is not Zobrist hashing as such.
User avatar
stegemma
Posts: 859
Joined: Mon Aug 10, 2009 10:05 pm
Location: Italy
Full name: Stefano Gemma

Re: Computing hash "on the fly"

Post by stegemma »

I've finally found a way to let it works, "crossing" the code with the great George Marsaglia random number generator:

Code: Select all

inline uint64_t uRandS&#40;uint64_t &seed0, uint64_t &seed1&#41;
	&#123;
		uint64_t x = seed0;
		uint64_t const y = seed1;
		seed0 = y;
		x ^= x << 23; // a
		x ^= x >> 17; // b
		x ^= y ^ &#40;y >> 26&#41;; // c
		seed1 = x;
		return x + y;
	&#125;

	uint64_t clsBoard&#58;&#58;GetHash&#40;)
	&#123;
		#if FAST_TT 
		uint64_t seeds&#91;&#93; = &#123; 0xA10E014701010401ULL, 0x02C070202CDE2ULL &#125;;
		#define HH&#40;a&#41; seeds&#91;0&#93; ^= a; boHash ^= uRandS&#40;seeds&#91;0&#93;, seeds&#91;1&#93;);
		boHash = boAllPieces;
			HH&#40;boKings&#41;;
			HH&#40;boQueens&#41;;
			HH&#40;boRooks&#41;;
			HH&#40;boBishops&#41;;
			HH&#40;boKnights&#41;;
			HH&#40;boPawns&#41;;
			HH&#40;color&#41;;
			HH&#40;boFriends&#41;;
			HH&#40;boEnpPawn - boCastlings&#41;;
			return boHash;
		#else 
			return boHash ^ boAllPieces ^ &#40;boEnpPawn - boCastlings&#41; ^ zobcol&#91;color&#93;;
		#endif 
	&#125;
This way I don't get false positive anymore and it has the same speed as the standard mode.

It must be further tested and optimized (I've tested only a few positions) but it could be a good solution for low memory systems (similar to my old "overlapping Zobrist key" post) or to limit memory access.
Author of Drago, Raffaela, Freccia, Satana, Sabrina.
http://www.linformatica.com
User avatar
stegemma
Posts: 859
Joined: Mon Aug 10, 2009 10:05 pm
Location: Italy
Full name: Stefano Gemma

Re: Computing hash "on the fly"

Post by stegemma »

My "standard" implementation is this one:

Code: Select all

clsTT *clsEngine&#58;&#58;GetTT&#40;uint64_t hash&#41;
&#123;
	clsTT *pTT = NULL;
	if&#40;nTT&#41;
	&#123;
		pTT = TT + &#40;hash % nTT&#41;; // (&#40;hash |&#40;hash>>32&#41;) % nTT&#41;;
	&#125;
	return pTT;
&#125;

inline uint64_t ZOB&#40;int c, int p, int i&#41;&#123;	return zob&#91;i&#93;&#91;p&#93;&#91;c&#93;; &#125;

void clsBoard&#58;&#58;SetZob&#40;int piece, int idx&#41; &#123; boHash ^= ZOB&#40;color, piece, idx&#41;; &#125;

void clsEngine&#58;&#58;ExecuteMove&#40;clsMove *pMove&#41;
&#123;
	BOMOVE&#40;pMove&#41;;
	#if ENABLE_TT && !FAST_TT
		int piece = pMove->boFlags & flg_pieceIdx;
		int idxSrc = GetSquareIndex&#40;boSrc&#41;;
		int idxDst = GetSquareIndex&#40;boDst&#41;;
		board.SetZob&#40;piece, idxSrc&#41;;
	#endif
...and other stuffs for captures etc...
&#125;

uint64_t clsBoard&#58;&#58;GetHash&#40;)
&#123;
	#if FAST_TT 
			uint64_t seeds&#91;&#93; = &#123; 0xA10E014701010401ULL, 0x02C070202CDE2ULL &#125;;
			#define HH&#40;a&#41; seeds&#91;0&#93; ^= &#40;a&#41;; boHash ^= uRandS&#40;seeds&#91;0&#93;, seeds&#91;1&#93;);
			boHash = boAllPieces;
			#if 0
				HH&#40;boKings-boQueens&#41;;
				HH&#40;boRooks-boBishops&#41;;
				HH&#40;boKnights-boPawns&#41;;
				HH&#40;boEnpPawn - boCastlings&#41;;
				HH&#40;boFriends&#41;;
				HH&#40;color&#41;;
			#else
				HH&#40;boKings&#41;;
				HH&#40;boQueens&#41;;
				HH&#40;boRooks&#41;;
				HH&#40;boBishops&#41;;
				HH&#40;boKnights&#41;;
				HH&#40;boPawns&#41;;
				HH&#40;color&#41;;
				HH&#40;boFriends&#41;;
				HH&#40;boEnpPawn - boCastlings&#41;;
			#endif
		return boHash;
	#else 
		return boHash ^ boAllPieces ^ &#40;boEnpPawn - boCastlings&#41; ^ zobcol&#91;color&#93;;
	#endif 
&#125;
In the new GetHash function I'm trying a faster mode (those with "HH(boKings-boQueens)") that it seems to works too (but it is more dangerous, I think). The Marsaglia code can be simplified, removing the "b" part or other parts but still then it needs more tests. All this changes get the nps increase, without changing the TT hits ratio.
Author of Drago, Raffaela, Freccia, Satana, Sabrina.
http://www.linformatica.com