Starting with Hash Tables.

Discussion of chess software programming and technical issues.

Moderators: hgm, Rebel, chrisw

kbhearn
Posts: 411
Joined: Thu Dec 30, 2010 4:48 am

Re: Starting with Hash Tables.

Post by kbhearn »

Alright, i think you have two sources of confusion:

1) The dim statement you provided earlier does not provide an array with multiple data entries per index, but an array with multiple indexes. It's neither a proper table nor an entry. We'll get back to the functional options for this in a moment.

2) The function of a transposition table and core parts
a) The hashed representation of your position
Your actual position representation is going to be much larger than the actual statespace of the game in order to be efficient to work with. positions in your search are going to be very similar such that if you use only a portion of your position representation you're going to get collisions. So you use a hashing function on the entirety of the game state. In chess engines, the established effective method for doing this is a zobrist hash key. For each possible state a 64-bit number is assigned (usually a random 64-bit number) - Then all of these 64-bit numbers are xored together, and the result is a position hash key that changing one thing on the board changes on average a random half of the bits. For your transposition table this resulting 64-bit key is what determines the index into your table, and what's not needed from that is used to determine when two positions that share the same index are not in fact the same position.

b) how the table is used
- at the end of your search for a node when you have a result, you convert the hash key into an index, look up that entry of the transposition table, and replace it (or one of it if using multiple buckets - but starting with single-bucket always-replace is probably simplest)
- at the top of the search for a node you then look up the index, check if the hash key matches what's found at that index and if so: check if the depth and entry type are sufficient to return a result now, if you can't take a return now from that then use the move stored first in the search for this node

c) back to defining the table - the right answer is to create a user-defined custom type composed of primitive types - but if that's something you want to avoid learning for now the alternative is multiple arrays - it's less efficient for many reasons but would look like this (note you may need to change some of these types for simplicity if for instance you're using floating point scores)

Code: Select all

Dim TTkey(tablesize-1) As ULongInt
Dim TTmove(tablesize-1) As UShort
Dim TTscore(tablesize-1) As Short
Dim TTdepth(tablesize-1) As Byte
Dim TTentrytype(tablesize-1) As UByte
I left out age for now since this is assuming single entry always replace.

creating a user defined type instead:

Code: Select all

Type TTEntry Field = 1
  key As ULongInt
  move As UShort
  score As Short
  depth As Byte
  entrytype As UByte
End Type

Dim TT(tablesize-1) As TTEntry
Edit: Added Byte-Alignment to type
Last edited by kbhearn on Sun Nov 13, 2016 10:08 pm, edited 1 time in total.
Sven
Posts: 4052
Joined: Thu May 15, 2008 9:57 pm
Location: Berlin, Germany
Full name: Sven Schüle

Re: Starting with Hash Tables.

Post by Sven »

Luis Babboni wrote:
Sven Schüle wrote: ...
Luis Babboni wrote:I use many other arrays in my code too.
I suggest to consider using user-defined data types (keyword TYPE in FreeBasic) where appropriate, instead of arrays. In fact it would be appropriate for many concepts in a chess engine like TTEntry, Move, Board, and some others. It would improve the program's readability and would simplify changing the code as well as debugging it.
I was just persuaded to do this, but need a complete rewrite of the code, so is scheduled for Soberango 1.xx.x :oops:
I think a rewrite is a good thing. Nevertheless, changing arrays into user-defined data types could also be done step by step, type by type.
Ras
Posts: 2487
Joined: Tue Aug 30, 2016 8:19 pm
Full name: Rasmus Althoff

Re: Starting with Hash Tables.

Post by Ras »

Btw, you can also have a separate hash table just for the pawn evaluation, that's what NG-Play had implemented when I came to it. What I added was entries for the rooks so that open/semi-open files are preferred, especially when targeting enemy backward pawns.
User avatar
Luis Babboni
Posts: 464
Joined: Sat Feb 28, 2015 4:37 pm
Location: Argentina

Re: Starting with Hash Tables.

Post by Luis Babboni »

hgm wrote:
Luis Babboni wrote:I still do not understand how the ruler could distingish if I´m using a TT or another kind of table to allow me to use for examepl just 256MB only for TT and more memory for others tables.

I think there is something I missing here. :(
Well, the protocol commands specify total memory that you are allowed to use for all tables together. It is sometimes assumed that all tables of significant size will be hash tables.

Unfortunately many engines cheat, and when the option controlling their tablesize is set to 128MB, would not hesitate to use 128MB just for their TT, and then another 600MB on top of it for, say, bitbases. If testers allow this kind of cheating is upto them.
Now is clear! Thanks H.G.! :D
User avatar
Luis Babboni
Posts: 464
Joined: Sat Feb 28, 2015 4:37 pm
Location: Argentina

Re: Starting with Hash Tables.

Post by Luis Babboni »

Thanks Kevin, studying your comments! :D
User avatar
Luis Babboni
Posts: 464
Joined: Sat Feb 28, 2015 4:37 pm
Location: Argentina

Re: Starting with Hash Tables.

Post by Luis Babboni »

Ras wrote:Btw, you can also have a separate hash table just for the pawn evaluation, that's what NG-Play had implemented when I came to it. What I added was entries for the rooks so that open/semi-open files are preferred, especially when targeting enemy backward pawns.
Thanks Rasmus, but still I far from that point of refination! :oops:
User avatar
Luis Babboni
Posts: 464
Joined: Sat Feb 28, 2015 4:37 pm
Location: Argentina

Re: Starting with Hash Tables.

Post by Luis Babboni »

Sven Schüle wrote:
Luis Babboni wrote:
Sven Schüle wrote: ...
Luis Babboni wrote:I use many other arrays in my code too.
I suggest to consider using user-defined data types (keyword TYPE in FreeBasic) where appropriate, instead of arrays. In fact it would be appropriate for many concepts in a chess engine like TTEntry, Move, Board, and some others. It would improve the program's readability and would simplify changing the code as well as debugging it.
I was just persuaded to do this, but need a complete rewrite of the code, so is scheduled for Soberango 1.xx.x :oops:
I think a rewrite is a good thing. Nevertheless, changing arrays into user-defined data types could also be done step by step, type by type.
Yes, could be possible.... but I´m terrified with the mistakes I will do doing it! :P
Sven
Posts: 4052
Joined: Thu May 15, 2008 9:57 pm
Location: Berlin, Germany
Full name: Sven Schüle

Re: Starting with Hash Tables.

Post by Sven »

Luis Babboni wrote:
Sven Schüle wrote:
Luis Babboni wrote:
Sven Schüle wrote: ...
Luis Babboni wrote:I use many other arrays in my code too.
I suggest to consider using user-defined data types (keyword TYPE in FreeBasic) where appropriate, instead of arrays. In fact it would be appropriate for many concepts in a chess engine like TTEntry, Move, Board, and some others. It would improve the program's readability and would simplify changing the code as well as debugging it.
I was just persuaded to do this, but need a complete rewrite of the code, so is scheduled for Soberango 1.xx.x :oops:
I think a rewrite is a good thing. Nevertheless, changing arrays into user-defined data types could also be done step by step, type by type.
Yes, could be possible.... but I´m terrified with the mistakes I will do doing it! :P
If you avoid making changes that improve software quality only due to the risk of introducing new errors then you keep the old structure which has even more potential to introduce errors with other changes. That sounds paradox. The best way would be - just do it right :D
User avatar
Luis Babboni
Posts: 464
Joined: Sat Feb 28, 2015 4:37 pm
Location: Argentina

Re: Starting with Hash Tables.

Post by Luis Babboni »

Sven Schüle wrote:
Luis Babboni wrote:
Sven Schüle wrote:
Luis Babboni wrote:
Sven Schüle wrote: ...
Luis Babboni wrote:I use many other arrays in my code too.
I suggest to consider using user-defined data types (keyword TYPE in FreeBasic) where appropriate, instead of arrays. In fact it would be appropriate for many concepts in a chess engine like TTEntry, Move, Board, and some others. It would improve the program's readability and would simplify changing the code as well as debugging it.
I was just persuaded to do this, but need a complete rewrite of the code, so is scheduled for Soberango 1.xx.x :oops:
I think a rewrite is a good thing. Nevertheless, changing arrays into user-defined data types could also be done step by step, type by type.
Yes, could be possible.... but I´m terrified with the mistakes I will do doing it! :P
If you avoid making changes that improve software quality only due to the risk of introducing new errors then you keep the old structure which has even more potential to introduce errors with other changes. That sounds paradox. The best way would be - just do it right :D
My actual target is not the software quality or even the engine strenght. My target is to try to understand the most important search strategies and fight a little with them to be able in the future to make better quality software and more strong engine rewriting it not every time I learn a new strategie.
For example, I have no move ordering yet the first time I see I need to rewrite the code.... but instead of rewrite the code, I could add move ordering to the engine without need it and now, when I´ll rewrite the code, I´ll could di it (I hope/think) better than what I could do before add move ordering.
May be at some point I´ll could not go ahead without rewrite the code, but actually I think I´ll could complete my basic engine before it. In my plan rest this about hash tables, some about to manage tablabases and openings and later some evaluation plus than my actual Shanon 9,5,3,3,1 material values.
I think in less than half of a year I´ll must be starting with a new code having a better idea about what I need than the idea I have now and sure far more idea than I had few months ago.

Of course I know my actual code is far to be a clean code, more than this is a really mess..... and is too possible that have many mistakes, but I think the fast road is learning about the issues I still do not even have an idea about how works (like table bases) before rewrite it...... is a thought of course, not sure of this, just my thought! :wink:
Sven
Posts: 4052
Joined: Thu May 15, 2008 9:57 pm
Location: Berlin, Germany
Full name: Sven Schüle

Re: Starting with Hash Tables.

Post by Sven »

Luis Babboni wrote:
Sven Schüle wrote:
Luis Babboni wrote:
Sven Schüle wrote:
Luis Babboni wrote:
Sven Schüle wrote: ...
Luis Babboni wrote:I use many other arrays in my code too.
I suggest to consider using user-defined data types (keyword TYPE in FreeBasic) where appropriate, instead of arrays. In fact it would be appropriate for many concepts in a chess engine like TTEntry, Move, Board, and some others. It would improve the program's readability and would simplify changing the code as well as debugging it.
I was just persuaded to do this, but need a complete rewrite of the code, so is scheduled for Soberango 1.xx.x :oops:
I think a rewrite is a good thing. Nevertheless, changing arrays into user-defined data types could also be done step by step, type by type.
Yes, could be possible.... but I´m terrified with the mistakes I will do doing it! :P
If you avoid making changes that improve software quality only due to the risk of introducing new errors then you keep the old structure which has even more potential to introduce errors with other changes. That sounds paradox. The best way would be - just do it right :D
My actual target is not the software quality or even the engine strenght. My target is to try to understand the most important search strategies and fight a little with them to be able in the future to make better quality software and more strong engine rewriting it not every time I learn a new strategie.
For example, I have no move ordering yet the first time I see I need to rewrite the code.... but instead of rewrite the code, I could add move ordering to the engine without need it and now, when I´ll rewrite the code, I´ll could di it (I hope/think) better than what I could do before add move ordering.
May be at some point I´ll could not go ahead without rewrite the code, but actually I think I´ll could complete my basic engine before it. In my plan rest this about hash tables, some about to manage tablabases and openings and later some evaluation plus than my actual Shanon 9,5,3,3,1 material values.
I think in less than half of a year I´ll must be starting with a new code having a better idea about what I need than the idea I have now and sure far more idea than I had few months ago.

Of course I know my actual code is far to be a clean code, more than this is a really mess..... and is too possible that have many mistakes, but I think the fast road is learning about the issues I still do not even have an idea about how works (like table bases) before rewrite it...... is a thought of course, not sure of this, just my thought! :wink:
Supporting endgame tablebases is not an important part of a chess engine. A proper structure of the program including a decent search is much more important.

I think you go the wrong road. Understanding chess programming concepts in theory is one step. But applying them in an own program is another step that requires proper knowledge about good programming in general. You have already admitted several times that you think your current code is "a mess", and you are planning a rewrite for a long time already. But you still add many basic features to that "mess", making it even more "messy". I hope you know what I mean. Each additional change becomes even more difficult, I don't know how you want to debug your program in that state.