No. For an extreme example, hash bigger than main memory will probably hurt.syzygy wrote:The only reason why bigger hash might not gain is that hash is already big enough (for a given test).
Too much Hash harms?
Moderators: hgm, Rebel, chrisw
-
- Posts: 2851
- Joined: Wed Mar 08, 2006 10:01 pm
- Location: Irvine, CA, USA
Re: Too much Hash harms?
Deasil is the right way to go.
-
- Posts: 10948
- Joined: Wed Jul 26, 2006 10:21 pm
- Full name: Kai Laskos
Re: Too much Hash harms?
Very interesting! Did you do testing on that? This 40% seems to be valid for Komodo, Stockfish and Houdini, so it's more general. There was some discussion about the usefulness of the Hash when it is too little, with estimation of say 5% time-to-depth speed-up for doubling Hash, but very few I can find on too much Hash, it even seemed that giving too much Hash wouldn't hurt at all.mjlef wrote:For years, Komodo has included a file setHash.txt which explains a process for a user to optimize the has sized for a specific machine and time control, and it in fact uses the 40% fill as the goal. Nice to confirm what we have found.Laskos wrote:
The extrapolation curve seems to indicate 32M as the optimum Hash and about 40% Hashfull. Also, too much Hash seems to harm as much as to little Hash (on logarithmic scale).
Mark
-
- Posts: 5960
- Joined: Sun Jan 10, 2010 6:15 am
- Location: Maryland USA
Re: Too much Hash harms?
The 40% figure dates back to before Mark joined the Komodo team, I think Don did most of the testing to determine this figure. In general at various time limits we have used hash sizes quite in line with the numbers you are coming up with. Back in my Rybka days Vas said that more Hash never hurts until you tax system resources, but that doesn't seem to be supported by the data.Laskos wrote:Very interesting! Did you do testing on that? This 40% seems to be valid for Komodo, Stockfish and Houdini, so it's more general. There was some discussion about the usefulness of the Hash when it is too little, with estimation of say 5% time-to-depth speed-up for doubling Hash, but very few I can find on too much Hash, it even seemed that giving too much Hash wouldn't hurt at all.mjlef wrote:For years, Komodo has included a file setHash.txt which explains a process for a user to optimize the has sized for a specific machine and time control, and it in fact uses the 40% fill as the goal. Nice to confirm what we have found.Laskos wrote:
The extrapolation curve seems to indicate 32M as the optimum Hash and about 40% Hashfull. Also, too much Hash seems to harm as much as to little Hash (on logarithmic scale).
Mark
Keep in mind that Komodo uses Hash sizes of 6,12,24,48 etc., not the usual 4,8,16,32 etc., so this can affect results. Since most (or all) testers choose Hash sizes that are a power of 2, we are always at a small disadvantage in those tests, probably just a couple elo or so. I think that our way is better for most users because people usually get a power of 2 of memory, and with our scheme they can leave just 25% for everything other than Hash instead of the usual 50%.
Komodo rules!
Re: Too much Hash harms?
Also:
So yeah you could say 32MB scores best but the error margin is quite a bit larger
Code: Select all
Rank Name Elo + - games score oppo. draws
1 32 1 6 6 5133 50% -1 53%
2 16 0 6 5 5133 50% 0 53%
3 64 -1 6 6 5134 50% 1 52%
-
- Posts: 219
- Joined: Wed Sep 19, 2007 11:07 am
- Location: Singapore
Re: Too much Hash harms?
Hi,
Mark in this thread mentioned that Komodo has included a file setHash.txt which explains a process for a user to optimize the hash size. The following is from setHash.txt:
"1. Take the main time in minutes and add the increment in seconds
2. Multiply the value obtained in step 1 by 3."
Does that mean that 40 moves in 180 minutes, we use 180 x 3 = 540MB for hash size?
Thank you & regards,
Koh, Kah Huat
Mark in this thread mentioned that Komodo has included a file setHash.txt which explains a process for a user to optimize the hash size. The following is from setHash.txt:
"1. Take the main time in minutes and add the increment in seconds
2. Multiply the value obtained in step 1 by 3."
Does that mean that 40 moves in 180 minutes, we use 180 x 3 = 540MB for hash size?
Thank you & regards,
Koh, Kah Huat
-
- Posts: 10948
- Joined: Wed Jul 26, 2006 10:21 pm
- Full name: Kai Laskos
Re: Too much Hash harms?
3. From the opening position, do a fixed time search of exactly this amount of time in seconds. (540 seconds in this example)Kohflote wrote:Hi,
Mark in this thread mentioned that Komodo has included a file setHash.txt which explains a process for a user to optimize the hash size. The following is from setHash.txt:
"1. Take the main time in minutes and add the increment in seconds
2. Multiply the value obtained in step 1 by 3."
Does that mean that 40 moves in 180 minutes, we use 180 x 3 = 540MB for hash size?
Thank you & regards,
Koh, Kah Huat
4. Note the hash table utilization as reported by the GUI.
5. If the hash table utilization is much above 40%, double the hash table size and repeat this test.
//// Be warned that say on 4 cores at this long time control the necessary Hash could be in 12-24GB range, which is probably higher than Hash size you imagine or allowed on your machine. ////
-
- Posts: 3293
- Joined: Wed Mar 08, 2006 8:15 pm
Re: Too much Hash harms?
From Houdini faq:
If you know the average move time T (in seconds) and the average node speed of
your hardware S (in kN/s), you can compute the optimal hash size with the formula: (T
x S / 100) MB.
That means in my corei5 that optimal for 1,5s/1core is 38MB, but for 60s/4cores already 6GB!
If you know the average move time T (in seconds) and the average node speed of
your hardware S (in kN/s), you can compute the optimal hash size with the formula: (T
x S / 100) MB.
That means in my corei5 that optimal for 1,5s/1core is 38MB, but for 60s/4cores already 6GB!
Jouni
Re: Too much Hash harms?
I'm starting to think it doesn't matter after a certain size. For my program that is:
Code: Select all
Rank Name Elo + - games score oppo. draws
1 dorpsgek 200 3 3 124380 79% -50 9%
2 64MB -48 5 5 31087 21% 200 9%
3 32MB -48 4 4 31091 21% 200 9%
4 16MB -49 5 5 31098 21% 200 9%
5 8MB -55 4 4 31104 21% 200 9%
-
- Posts: 10948
- Joined: Wed Jul 26, 2006 10:21 pm
- Full name: Kai Laskos
Re: Too much Hash harms?
Well, the error margins are still large, and the Hash span is not that large, try up to 1GB or so. Also, it's more efficient and easier to check time-to-depth or even NPS for too much hash than games, which are unnecessarily CPU time consuming.flok wrote:I'm starting to think it doesn't matter after a certain size. For my program that is:
Code: Select all
Rank Name Elo + - games score oppo. draws 1 dorpsgek 200 3 3 124380 79% -50 9% 2 64MB -48 5 5 31087 21% 200 9% 3 32MB -48 4 4 31091 21% 200 9% 4 16MB -49 5 5 31098 21% 200 9% 5 8MB -55 4 4 31104 21% 200 9%
-
- Posts: 5566
- Joined: Tue Feb 28, 2012 11:56 pm
Re: Too much Hash harms?
You removed my statement that it is the nps drop that hurts.Dirt wrote:No. For an extreme example, hash bigger than main memory will probably hurt.syzygy wrote:The only reason why bigger hash might not gain is that hash is already big enough (for a given test).
Hash bigger than main memory will lead to a very substantial nps drop and hurts for that reason and only for that reason.
Again: correct for nps drop by looking only at total nodes searched, and any disadvantage of bigger hash disappears. The one and only disadvantage of bigger hash is the nps drop.
Enabling large pages (i.e. make proper use of the hardware) significantly reduces the nps drop.