Too much Hash harms?

Laskos · Post by **Laskos** » Sun Feb 05, 2017 10:05 am

Adam Hair wrote:
Laskos wrote:
hgm wrote:
IIRC in my measurements the search time only started to go up once the overload factor exeded 10.
That seems confirmed. In 1000 games each, about 60,000 positions per Stockfish with different Hash at 6''+0.06'' 1 core, I get the following depths and nps:
Code: Select all
 1.  SF 1M         tpm=148.0 d=16.14 nps=1951575    99%
 2.  SF 4M         tpm=148.1 d=16.30 nps=1955569    80%
 3.  SF 16M        tpm=149.3 d=16.36 nps=1946288    25%
 4.  SF 64M        tpm=149.3 d=16.33 nps=1910916     7%
 5.  SF 256M       tpm=149.5 d=16.25 nps=1817734     2%
 6.  SF 1024M      tpm=149.4 d=16.14 nps=1718677     1%
The last column is approximate Hash usage per move out of available. 25% (16M Hash) seems the optimum here depth-wise. And indeed, a factor of 16 overload seems to not hurt almost at all (4M vs 64M).
I measured depth, nodes per second, and hash usage at 1.5 seconds per move from a set of positions (the 8238 positions from the sim tool; data extracted from Polyglot logs). I think the thinking time would equate to 1 to 1.2 seconds per move on your computer.
Code: Select all
  
Hash	Depth	   NPS	    Hashfull
   1	19.71	1423911.54	98.92%
   4	19.84	1423019.10	98.59%
  16	19.99	1471216.76	73.69%
  32	19.90	1425357.18	43.74%
  64	19.87	1407208.73	22.82%
 256	19.84	1385323.14	 5.69%
1024	19.71	1310237.31	 2.06%
4096	19.66	1284870.14	 1.65%
16 MB of hash seems to be best for this tpm and these positions on my computer. I was expecting a little more of a difference compared to your results.

Good, your Hashfull is very useful, I have just estimation by looking at Hash usage for typical positions. I refined my test now, in LittleBlitzer I just play for 5 moves from 2moves_v1.epd openings, so only opening positions with similar time-to-depth. With InBetween I let them play at fixed depth=17 (about one second per move, but a bit varied time). The result is here, and is more precise than my previous one, because time-to-depth variance is smaller than using full games.

Code: Select all

Games Completed = 1800 of 1800 &#40;Avg game length = 9.963 sec&#41;
Settings = RR/0MB/100000ms per move/M 500cp for 2 moves, D 5 moves/EPD&#58;C&#58;\LittleBlitzer\2moves_v1.epd&#40;32000&#41;
Time = 5544 sec elapsed, 0 sec remaining
 1.  SF 1M                    	300.0/600	0-0-600  	&#40;tpm=964.0 d=17.00 nps=1367595&#41;  99%
 2.  SF 4M                    	300.0/600	0-0-600  	&#40;tpm=901.0 d=17.00 nps=1376814&#41;  98%
 3.  SF 16M                   	300.0/600	0-0-600  	&#40;tpm=860.5 d=17.00 nps=1383201&#41;  60%
 4.  SF 64M                   	300.0/600	0-0-600  	&#40;tpm=861.7 d=17.00 nps=1373527&#41;  18%
 5.  SF 256M                  	300.0/600	0-0-600  	&#40;tpm=901.5 d=17.00 nps=1329987&#41;   5%
 6.  SF 1024M                 	300.0/600	0-0-600  	&#40;tpm=945.3 d=17.00 nps=1255824&#41;   2%

The Hashfull in the last column is an estimation. Your numbers on this are more reliable. The plot of time-to-depth=17 for those opening positions is here:

The extrapolation curve seems to indicate 32M as the optimum Hash and about 40% Hashfull. Also, too much Hash seems to harm as much as to little Hash (on logarithmic scale).

Jouni · Post by **Jouni** » Sun Feb 05, 2017 10:25 am

I assume, that tests were done with SF? In my tests Houdini and Komodo seems to behave differently: they prefer bigger hash!?

Laskos · Post by **Laskos** » Sun Feb 05, 2017 10:40 am

Jouni wrote:I assume, that tests were done with SF? In my tests Houdini and Komodo seems to behave differently: they prefer bigger hash!?

Yes, SF. Seems weird if what you say is true.

syzygy · Post by **syzygy** » Sun Feb 05, 2017 11:13 am

Laskos wrote:Also, too much Hash seems to harm as much as to little Hash (on logarithmic scale).

Too much hash can harm only if the decrease in nps outweighs the gain of bigger hash.

So if you correct for nps decrease (so just look at total node count), the harm will disappear completely (modulo statistical noise).

The only reason why bigger hash might not gain is that hash is already big enough (for a given test).

Laskos · Post by **Laskos** » Sun Feb 05, 2017 11:22 am

syzygy wrote:
Laskos wrote:Also, too much Hash seems to harm as much as to little Hash (on logarithmic scale).
Too much hash can harm only if the decrease in nps outweighs the gain of bigger hash.

So if you correct for nps decrease (so just look at total node count), the harm will disappear completely (modulo statistical noise).

The only reason why bigger hash might not gain is that hash is already big enough (for a given test).

Yes, NPS drop is substantial after the sweet spot, although the sweet spot in NPS is not necessarily equal in Hash to the sweet spot in time-to-depth, but they seem to be close.

Laskos · Post by **Laskos** » Sun Feb 05, 2017 1:09 pm

Laskos wrote:
Jouni wrote:I assume, that tests were done with SF? In my tests Houdini and Komodo seems to behave differently: they prefer bigger hash!?
Yes, SF. Seems weird if what you say is true.

Seems not to be true for Komodo 10.3. Time-to-depth=17 are here, and their succession seems similar to SF:

Code: Select all

Games Completed = 1800 of 1800 &#40;Avg game length = 11.923 sec&#41;
Settings = RR/0MB/100000ms per move/M 500cp for 2 moves, D 5 moves/EPD&#58;C&#58;\LittleBlitzer\2moves_v1.epd&#40;32000&#41;
Time = 6469 sec elapsed, 0 sec remaining
 1.  K 1M                     	300.0/600	0-0-600  	&#40;tpm=1104.4 d=17.00 nps=1234770&#41;  99%
 2.  K 3M                     	300.0/600	0-0-600  	&#40;tpm=1080.4 d=17.00 nps=1224209&#41;  98%
 3.  K 12M                    	300.0/600	0-0-600  	&#40;tpm=1071.3 d=17.00 nps=1212606&#41;  70%
 4.  K 48M                    	300.0/600	0-0-600  	&#40;tpm=1063.3 d=17.00 nps=1210377&#41;  20%
 5.  K 192M                   	300.0/600	0-0-600  	&#40;tpm=1067.3 d=17.00 nps=1185387&#41;   5%
 6.  K 768M                  	 300.0/600	0-0-600  	&#40;tpm=1136.6 d=17.00 nps=1139547&#41;   2%

It seems that 48M is optimal, comparable to SF 32M. Time-to-depth=17 is a bit longer for Komodo, so the result is normal.

flok · Post by **flok** » Sun Feb 05, 2017 3:06 pm

Odd: for Embla there is no difference at all. That is: no TT is dramatic but 1MB or 2048MB no difference:

Code: Select all

  rev 3799
	1 info string nps&#58; 242k, eval&#58; 7, depth&#58; 14, time&#58; 9.428
	2 info string nps&#58; 240k, eval&#58; 7, depth&#58; 14, time&#58; 9.496
	4 info string nps&#58; 243k, eval&#58; 7, depth&#58; 14, time&#58; 9.385
	8 info string nps&#58; 237k, eval&#58; 7, depth&#58; 14, time&#58; 9.614
	16 info string nps&#58; 242k, eval&#58; 7, depth&#58; 14, time&#58; 9.399
	32 info string nps&#58; 243k, eval&#58; 7, depth&#58; 14, time&#58; 9.376
	64 info string nps&#58; 243k, eval&#58; 7, depth&#58; 14, time&#58; 9.370
	128 info string nps&#58; 243k, eval&#58; 7, depth&#58; 14, time&#58; 9.391
	256 info string nps&#58; 243k, eval&#58; 7, depth&#58; 14, time&#58; 9.388
	512 info string nps&#58; 236k, eval&#58; 7, depth&#58; 14, time&#58; 9.655
	1024 info string nps&#58; 243k, eval&#58; 7, depth&#58; 14, time&#58; 9.385
	2048 info string nps&#58; 242k, eval&#58; 7, depth&#58; 14, time&#58; 9.420
	4096 info string nps&#58; 240k, eval&#58; 7, depth&#58; 14, time&#58; 9.476

	without tt&#58;
	x info string nps&#58; 256k, eval&#58; 7, depth&#58; 14, time&#58; 67.545

Laskos · Post by **Laskos** » Sun Feb 05, 2017 3:22 pm

flok wrote:Odd: for Embla there is no difference at all. That is: no TT is dramatic but 1MB or 2048MB no difference:

Code: Select all

  rev 3799
	1 info string nps&#58; 242k, eval&#58; 7, depth&#58; 14, time&#58; 9.428
	2 info string nps&#58; 240k, eval&#58; 7, depth&#58; 14, time&#58; 9.496
	4 info string nps&#58; 243k, eval&#58; 7, depth&#58; 14, time&#58; 9.385
	8 info string nps&#58; 237k, eval&#58; 7, depth&#58; 14, time&#58; 9.614
	16 info string nps&#58; 242k, eval&#58; 7, depth&#58; 14, time&#58; 9.399
	32 info string nps&#58; 243k, eval&#58; 7, depth&#58; 14, time&#58; 9.376
	64 info string nps&#58; 243k, eval&#58; 7, depth&#58; 14, time&#58; 9.370
	128 info string nps&#58; 243k, eval&#58; 7, depth&#58; 14, time&#58; 9.391
	256 info string nps&#58; 243k, eval&#58; 7, depth&#58; 14, time&#58; 9.388
	512 info string nps&#58; 236k, eval&#58; 7, depth&#58; 14, time&#58; 9.655
	1024 info string nps&#58; 243k, eval&#58; 7, depth&#58; 14, time&#58; 9.385
	2048 info string nps&#58; 242k, eval&#58; 7, depth&#58; 14, time&#58; 9.420
	4096 info string nps&#58; 240k, eval&#58; 7, depth&#58; 14, time&#58; 9.476

	without tt&#58;
	x info string nps&#58; 256k, eval&#58; 7, depth&#58; 14, time&#58; 67.545

Yes, seems odd, especially the more stable NPS here. How many positions did you use?

mjlef · Post by **mjlef** » Sun Feb 05, 2017 9:03 pm

Laskos wrote:
Adam Hair wrote:
Laskos wrote:
hgm wrote:
IIRC in my measurements the search time only started to go up once the overload factor exeded 10.
That seems confirmed. In 1000 games each, about 60,000 positions per Stockfish with different Hash at 6''+0.06'' 1 core, I get the following depths and nps:
Code: Select all
 1.  SF 1M         tpm=148.0 d=16.14 nps=1951575    99%
 2.  SF 4M         tpm=148.1 d=16.30 nps=1955569    80%
 3.  SF 16M        tpm=149.3 d=16.36 nps=1946288    25%
 4.  SF 64M        tpm=149.3 d=16.33 nps=1910916     7%
 5.  SF 256M       tpm=149.5 d=16.25 nps=1817734     2%
 6.  SF 1024M      tpm=149.4 d=16.14 nps=1718677     1%
The last column is approximate Hash usage per move out of available. 25% (16M Hash) seems the optimum here depth-wise. And indeed, a factor of 16 overload seems to not hurt almost at all (4M vs 64M).
I measured depth, nodes per second, and hash usage at 1.5 seconds per move from a set of positions (the 8238 positions from the sim tool; data extracted from Polyglot logs). I think the thinking time would equate to 1 to 1.2 seconds per move on your computer.
Code: Select all
  
Hash	Depth	   NPS	    Hashfull
   1	19.71	1423911.54	98.92%
   4	19.84	1423019.10	98.59%
  16	19.99	1471216.76	73.69%
  32	19.90	1425357.18	43.74%
  64	19.87	1407208.73	22.82%
 256	19.84	1385323.14	 5.69%
1024	19.71	1310237.31	 2.06%
4096	19.66	1284870.14	 1.65%
16 MB of hash seems to be best for this tpm and these positions on my computer. I was expecting a little more of a difference compared to your results.
Good, your Hashfull is very useful, I have just estimation by looking at Hash usage for typical positions. I refined my test now, in LittleBlitzer I just play for 5 moves from 2moves_v1.epd openings, so only opening positions with similar time-to-depth. With InBetween I let them play at fixed depth=17 (about one second per move, but a bit varied time). The result is here, and is more precise than my previous one, because time-to-depth variance is smaller than using full games.
Code: Select all
Games Completed = 1800 of 1800 &#40;Avg game length = 9.963 sec&#41;
Settings = RR/0MB/100000ms per move/M 500cp for 2 moves, D 5 moves/EPD&#58;C&#58;\LittleBlitzer\2moves_v1.epd&#40;32000&#41;
Time = 5544 sec elapsed, 0 sec remaining
 1.  SF 1M                    	300.0/600	0-0-600  	&#40;tpm=964.0 d=17.00 nps=1367595&#41;  99%
 2.  SF 4M                    	300.0/600	0-0-600  	&#40;tpm=901.0 d=17.00 nps=1376814&#41;  98%
 3.  SF 16M                   	300.0/600	0-0-600  	&#40;tpm=860.5 d=17.00 nps=1383201&#41;  60%
 4.  SF 64M                   	300.0/600	0-0-600  	&#40;tpm=861.7 d=17.00 nps=1373527&#41;  18%
 5.  SF 256M                  	300.0/600	0-0-600  	&#40;tpm=901.5 d=17.00 nps=1329987&#41;   5%
 6.  SF 1024M                 	300.0/600	0-0-600  	&#40;tpm=945.3 d=17.00 nps=1255824&#41;   2%
The Hashfull in the last column is an estimation. Your numbers on this are more reliable. The plot of time-to-depth=17 for those opening positions is here:

The extrapolation curve seems to indicate 32M as the optimum Hash and about 40% Hashfull. Also, too much Hash seems to harm as much as to little Hash (on logarithmic scale).

For years, Komodo has included a file setHash.txt which explains a process for a user to optimize the has sized for a specific machine and time control, and it in fact uses the 40% fill as the goal. Nice to confirm what we have found.

Mark

flok · Post by **flok** » Sun Feb 05, 2017 9:14 pm

Laskos wrote:Yes, seems odd, especially the more stable NPS here. How many positions did you use?

My test was very much broken

I used the same hash-size and only 1 position.
The following results are for 327 positions each:

Code: Select all

1 &#123; "skipped" &#58; 3, "found" &#58; 272, "total" &#58; 327, "took" &#58; 54.711000, "version" &#58; "0.9.8 ponder", "avg-nps" &#58; 284010.052823 &#125;
2 &#123; "skipped" &#58; 3, "found" &#58; 272, "total" &#58; 327, "took" &#58; 51.638000, "version" &#58; "0.9.8 ponder", "avg-nps" &#58; 277504.028041 &#125;
4 &#123; "skipped" &#58; 3, "found" &#58; 272, "total" &#58; 327, "took" &#58; 52.123000, "version" &#58; "0.9.8 ponder", "avg-nps" &#58; 277093.202617 &#125;
8 &#123; "skipped" &#58; 3, "found" &#58; 274, "total" &#58; 327, "took" &#58; 51.963000, "version" &#58; "0.9.8 ponder", "avg-nps" &#58; 270581.586898 &#125;
16 &#123; "skipped" &#58; 3, "found" &#58; 273, "total" &#58; 327, "took" &#58; 51.012000, "version" &#58; "0.9.8 ponder", "avg-nps" &#58; 266421.038187 &#125;
32 &#123; "skipped" &#58; 3, "found" &#58; 272, "total" &#58; 327, "took" &#58; 50.852000, "version" &#58; "0.9.8 ponder", "avg-nps" &#58; 270994.533155 &#125;
64 &#123; "skipped" &#58; 3, "found" &#58; 272, "total" &#58; 327, "took" &#58; 51.689000, "version" &#58; "0.9.8 ponder", "avg-nps" &#58; 269201.570934 &#125;
128 &#123; "skipped" &#58; 3, "found" &#58; 271, "total" &#58; 327, "took" &#58; 52.512000, "version" &#58; "0.9.8 ponder", "avg-nps" &#58; 268882.312614 &#125;
256 &#123; "skipped" &#58; 3, "found" &#58; 271, "total" &#58; 327, "took" &#58; 51.712000, "version" &#58; "0.9.8 ponder", "avg-nps" &#58; 268210.009282 &#125;
512 &#123; "skipped" &#58; 3, "found" &#58; 271, "total" &#58; 327, "took" &#58; 52.265000, "version" &#58; "0.9.8 ponder", "avg-nps" &#58; 264162.307472 &#125; 
1024 &#123; "skipped" &#58; 3, "found" &#58; 271, "total" &#58; 327, "took" &#58; 51.576000, "version" &#58; "0.9.8 ponder", "avg-nps" &#58; 261929.676594 &#125;
2048 &#123; "skipped" &#58; 3, "found" &#58; 271, "total" &#58; 327, "took" &#58; 52.174000, "version" &#58; "0.9.8 ponder", "avg-nps" &#58; 261606.010657 &#125;
4096 &#123; "skipped" &#58; 3, "found" &#58; 271, "total" &#58; 327, "took" &#58; 52.871000, "version" &#58; "0.9.8 ponder", "avg-nps" &#58; 256738.155132 &#125;

And altough the nps is not the highest (only 270k where the highest is 284k nps), it processed the test-file in the shortest time (50.9s, longest is 54.7s) for 32MB hash.

Too much Hash harms?

Re: Too much Hash harms?

Re: Too much Hash harms?

Re: Too much Hash harms?

Re: Too much Hash harms?

Re: Too much Hash harms?

Re: Too much Hash harms?

Re: Too much Hash harms?

Re: Too much Hash harms?

Re: Too much Hash harms?

Re: Too much Hash harms?