RJN wrote:Does anyone else see hash size for Komodo 9.1 being 1-2GB smaller when looking at what is actually being reported with task manager?
With 4GB hash, I see 3GB reported as used.
With 8GB hash, I see 6GB reported as used.
(I did not try other sizes yet)
Komodo 9.1 Win7 64 bit
That is normal. I redesigned the hash system to fit in about 75% of the space it took up before. When you tell the computer to use say 8 GB of memory, Komodo will allocate the multiple of 2 for the biggest hash table it can fit in that space. Due to the way hashing works, each increase in size is a doubling. So anyway, 6 GB is the biggest hash it can allocate in an 8 GB chunk of memory. The next step up would be 12 GB, and so on. Doing this let us fit in more hashed positions in a given amount of memory. Basically you are getting what was 8 GB of memory now compacted to only take up 6 GB.
I wonder if you achieved this by reducing the size of hash entries by 1/3. Doesn't that increase the chance of hash collisions?
The smaller you make the "key" part of the hash entry, the greater the risk of "collisions" which are entries that might falsely match the position. Some programs use very tiny keys, and depend on using another part of the key as a index to the position. I have seen some program use 16 bits for this key, and experiments using even less. Surprisingly, even the tiny keys do not seem to cause many collisions, and even if they collide, few of these collisions are bad enough to change the root score in a game result changing way.
Bob Hyatt and Anthony Cozzie wrote a very good paper on this:
cma6 wrote:LK:
Thank you for your continued rapid development of Komodo.
Unfortunately, although I bought version 9.0, there is no online way to upgrade to 9.1 without paying another $99. I hope you can iron out the glitches in the online upgrade process.
On happier note, let me recommend LK's "Sabotage the Gruenfeld", which I have just gone through page by page and line by line. It is the first GM opening book to fully use computer evals with GM insight to create part of your opening repertoire.
You should be able to purchase version 9.1 for $47.98 reflecting the 20% discount for K9 buyers. Others have done so. Let me know if you can't do this.
cma6 wrote:Ted,
Thanks for your rapid response, greatly appreciated.
I have emailed Mark but no response there.
I should have been clearer, I meant the book. Glad to see Larry responded in any event.
"Good decisions come from experience, and experience comes from bad decisions."
__________________________________________________________________
Ted Summers
cma6 wrote:LK:
Thank you for your continued rapid development of Komodo.
Unfortunately, although I bought version 9.0, there is no online way to upgrade to 9.1 without paying another $99. I hope you can iron out the glitches in the online upgrade process.
On happier note, let me recommend LK's "Sabotage the Gruenfeld", which I have just gone through page by page and line by line. It is the first GM opening book to fully use computer evals with GM insight to create part of your opening repertoire.
People who always want the bleeding edge version should get the "give me every bleeding edge update" plan.
RJN wrote:Does anyone else see hash size for Komodo 9.1 being 1-2GB smaller when looking at what is actually being reported with task manager?
With 4GB hash, I see 3GB reported as used.
With 8GB hash, I see 6GB reported as used.
(I did not try other sizes yet)
Komodo 9.1 Win7 64 bit
That is normal. I redesigned the hash system to fit in about 75% of the space it took up before. When you tell the computer to use say 8 GB of memory, Komodo will allocate the multiple of 2 for the biggest hash table it can fit in that space. Due to the way hashing works, each increase in size is a doubling. So anyway, 6 GB is the biggest hash it can allocate in an 8 GB chunk of memory. The next step up would be 12 GB, and so on. Doing this let us fit in more hashed positions in a given amount of memory. Basically you are getting what was 8 GB of memory now compacted to only take up 6 GB.
I wonder if you achieved this by reducing the size of hash entries by 1/3. Doesn't that increase the chance of hash collisions?
The smaller you make the "key" part of the hash entry, the greater the risk of "collisions" which are entries that might falsely match the position. Some programs use very tiny keys, and depend on using another part of the key as a index to the position. I have seen some program use 16 bits for this key, and experiments using even less. Surprisingly, even the tiny keys do not seem to cause many collisions, and even if they collide, few of these collisions are bad enough to change the root score in a game result changing way.
Bob Hyatt and Anthony Cozzie wrote a very good paper on this:
I know Stockfish reduced there hash key size a while ago, and there is probably some good data about this on the stockfish forum.
In the future, we might have to rethink things as machines get faster and faster.
Remember that with a 64 bit hash signature, if you store 16, you are not really using a 16 bit signature, you are using something significantly larger since you use the other part of the signature to address the table. In the current version of Crafty, I have moved to an 8 byte hash entry. 23 bits for the upper part of the signature. But when I run with a 32gb hash table, that turns into 4 billion entries (2^32). Which means I use the rightmost 32 bits of the entry to address the table, and store the leftmost 23 bits. That only leaves 9 bits unaccounted for out of the original 64. An advantage of this is that you can use a hash bucket size of 8, since that is still exactly one cache block (if you align things correctly of course) which gives you a better chance to replace something useless rather than something good.
I have tested this to exhaustion with zero problems. Even when searching at 100M nodes per second, giving billions of nodes even for pretty fast games.
I should have been clearer, I meant the book. Glad to see Larry responded in any event.