Table Base Cache Size question

Discussion of anything and everything relating to chess playing software and machines.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
User avatar
RJN
Posts: 303
Joined: Fri Jun 21, 2013 3:18 am
Location: Orion Spiral Arm

Table Base Cache Size question

Post by RJN » Thu Dec 05, 2013 3:53 am

Is there any downside to setting the endgame tablebase cache size to maximum on a machine with plenty of RAM to spare?

I ask because I want to make my tests as accurate and fair as possible, and I am running engines in a tournament comparing different TB performances. Syzygy has the ability to cache in the OS, which is much larger than any of the other EGTB cache limits of 1 GB maximum.

Will performance suffer if I set Gaviota or Nalimov cache to maximum? Note: I restart the engines for each game.

(and if so, why? Does the entire cache need to be searched for stored results, even empty portions of it?)

TIA

User avatar
michiguel
Posts: 6401
Joined: Thu Mar 09, 2006 7:30 pm
Location: Chicago, Illinois, USA
Contact:

Re: Table Base Cache Size question

Post by michiguel » Thu Dec 05, 2013 2:06 pm

RJN wrote:Is there any downside to setting the endgame tablebase cache size to maximum on a machine with plenty of RAM to spare?

I ask because I want to make my tests as accurate and fair as possible, and I am running engines in a tournament comparing different TB performances. Syzygy has the ability to cache in the OS, which is much larger than any of the other EGTB cache limits of 1 GB maximum.

Will performance suffer if I set Gaviota or Nalimov cache to maximum? Note: I restart the engines for each game.

(and if so, why? Does the entire cache need to be searched for stored results, even empty portions of it?)

TIA
You are addressing a very good point. The way memory is used make the comparisons difficult.

Gaviota TBs were not optimized to be that big (for the simple reason it would be better to have that memory used as hashtable and I never give more than 5% of HT) but it may be ok. I should look at this, since now humongous amount of memory are more available.

Miguel

User avatar
RJN
Posts: 303
Joined: Fri Jun 21, 2013 3:18 am
Location: Orion Spiral Arm

Re: Table Base Cache Size question

Post by RJN » Thu Dec 05, 2013 9:25 pm

Thanks for the reply, Miguel.

I have tried searching for an answer to the pros and cons of setting the EGTB cache "properly", but found little material, except what you pointed out, it could waste RAM. But with memory plentiful, I am curious what bad effects would arise from having the cache large. Does it only waste RAM, or does having a large cache noticeably lower the efficiency of the cache? (memory cache is still better than disk or even SSD access, I would think)

I have a 16GB system, so the largest hash I can use is 8GB, due to system overhead. This means I have plenty of RAM left for cache, yet most engines limit the cache to 1GB or less. Why this limit? I have done long analysis of endgames and seen the cache eventually fill, while wondering why it is limited to some seemingly arbitrary value. Why not a limit of, for example, 4GB?

I am curious what other authors have to say on this subject.

bob
Posts: 20916
Joined: Mon Feb 27, 2006 6:30 pm
Location: Birmingham, AL

Re: Table Base Cache Size question

Post by bob » Fri Dec 06, 2013 4:19 pm

RJN wrote:Thanks for the reply, Miguel.

I have tried searching for an answer to the pros and cons of setting the EGTB cache "properly", but found little material, except what you pointed out, it could waste RAM. But with memory plentiful, I am curious what bad effects would arise from having the cache large. Does it only waste RAM, or does having a large cache noticeably lower the efficiency of the cache? (memory cache is still better than disk or even SSD access, I would think)

I have a 16GB system, so the largest hash I can use is 8GB, due to system overhead. This means I have plenty of RAM left for cache, yet most engines limit the cache to 1GB or less. Why this limit? I have done long analysis of endgames and seen the cache eventually fill, while wondering why it is limited to some seemingly arbitrary value. Why not a limit of, for example, 4GB?

I am curious what other authors have to say on this subject.
If you have lots of memory that is unused, it becomes cache ANYWAY. The O/S will cache as many blocks of files as it can. In the case of Nalimov, the only issue is that the filesystem caches compressed blocks, while the egtb cache holds uncompressed blocks, making those a bit quicker to access. But either way, I/O will be minimized if you have the memory, unless you eat it all up with hash table...

User avatar
RJN
Posts: 303
Joined: Fri Jun 21, 2013 3:18 am
Location: Orion Spiral Arm

Re: Table Base Cache Size question

Post by RJN » Wed Dec 18, 2013 8:42 pm

bob wrote:
RJN wrote:Thanks for the reply, Miguel.

I have tried searching for an answer to the pros and cons of setting the EGTB cache "properly", but found little material, except what you pointed out, it could waste RAM. But with memory plentiful, I am curious what bad effects would arise from having the cache large. Does it only waste RAM, or does having a large cache noticeably lower the efficiency of the cache? (memory cache is still better than disk or even SSD access, I would think)

I have a 16GB system, so the largest hash I can use is 8GB, due to system overhead. This means I have plenty of RAM left for cache, yet most engines limit the cache to 1GB or less. Why this limit? I have done long analysis of endgames and seen the cache eventually fill, while wondering why it is limited to some seemingly arbitrary value. Why not a limit of, for example, 4GB?

I am curious what other authors have to say on this subject.
If you have lots of memory that is unused, it becomes cache ANYWAY. The O/S will cache as many blocks of files as it can. In the case of Nalimov, the only issue is that the filesystem caches compressed blocks, while the egtb cache holds uncompressed blocks, making those a bit quicker to access. But either way, I/O will be minimized if you have the memory, unless you eat it all up with hash table...
Thanks, I take this to mean that on a 16 GB machine running a tournament with 512 MB hash, there is basically no downside to setting the TB cache to maximum of 1 GB.

I am only using Gaviota and Syzygy TBs in the tournament, no Nalimov. Syzygy, of course, has no cache parameter. I was thinking that setting the Gaviota cache high probably does not help too much during a 40 in 5 minute game, since the engines rarely uses more than 256MB Gaviota cache at that time control (even with low probe depths) but it would at least possibly limit Syzygy RAM availability a bit, making things fairer. Syzygy (especially SF Syzygy) gets MUCH higher TB hit counts than Gaviota, even with the probe depths for Gaviota set low. Also, I am using 6-piece Syzygy, which triggers earlier TB access.

For analysis though, I wish there was not a 1GB limit for Gaviota cache.
Last edited by RJN on Wed Dec 18, 2013 8:50 pm, edited 1 time in total.

mvk
Posts: 589
Joined: Tue Jun 04, 2013 8:15 pm

Re: Table Base Cache Size question

Post by mvk » Wed Dec 18, 2013 8:50 pm

Setting the Gaviota cache very high might impact the Syzygy engines negatively. You need to reserve a healthy amount of OS cache for those.
[Account deleted]

User avatar
Mike S.
Posts: 1480
Joined: Thu Mar 09, 2006 4:33 am

Re: Table Base Cache Size question

Post by Mike S. » Wed Dec 18, 2013 8:52 pm

The guideline I always recommend is:

Maximum is NOT optimum. :mrgreen:

But I do not have data to back it up...
Regards, Mike

User avatar
RJN
Posts: 303
Joined: Fri Jun 21, 2013 3:18 am
Location: Orion Spiral Arm

Re: Table Base Cache Size question

Post by RJN » Wed Dec 18, 2013 8:55 pm

mvk wrote:Setting the Gaviota cache very high might impact the Syzygy engines negatively. You need to reserve a healthy amount of OS cache for those.
I have more than 12GB physical free for Syzygy (OS) use during the tournament, so I don't think Syzygy would suffer too much +/- 1GB, no?

Probably an academic question thread, I was just wondering.

mvk
Posts: 589
Joined: Tue Jun 04, 2013 8:15 pm

Re: Table Base Cache Size question

Post by mvk » Wed Dec 18, 2013 8:59 pm

RJN wrote:
mvk wrote:Setting the Gaviota cache very high might impact the Syzygy engines negatively. You need to reserve a healthy amount of OS cache for those.
I have more than 12GB physical free for Syzygy (OS) use during the tournament, so I don't think Syzygy would suffer too much +/- 1GB, no?

Probably an academic question thread, I was just wondering.
12GB (or 11GB) certainly seems sufficient. What also matters is if there is one or more games playing.
[Account deleted]

ThomasJMiller
Posts: 96
Joined: Fri Jan 11, 2013 11:22 am

Re: Table Base Cache Size question

Post by ThomasJMiller » Thu Dec 19, 2013 10:42 am

a slightly offtopic quesiton: I'm thinking of buying a SSD for SyzygyBases. What specs should I take in consideration besides size of course?
I think IOPS value is important?

Post Reply