Table Base Cache Size question

[RJN]

Is there any downside to setting the endgame tablebase cache size to maximum on a machine with plenty of RAM to spare?

I ask because I want to make my tests as accurate and fair as possible, and I am running engines in a tournament comparing different TB performances. Syzygy has the ability to cache in the OS, which is much larger than any of the other EGTB cache limits of 1 GB maximum.

Will performance suffer if I set Gaviota or Nalimov cache to maximum? Note: I restart the engines for each game.

(and if so, why? Does the entire cache need to be searched for stored results, even empty portions of it?)

TIA

[michiguel]

Is there any downside to setting the endgame tablebase cache size to maximum on a machine with plenty of RAM to spare?

I ask because I want to make my tests as accurate and fair as possible, and I am running engines in a tournament comparing different TB performances. Syzygy has the ability to cache in the OS, which is much larger than any of the other EGTB cache limits of 1 GB maximum.

Will performance suffer if I set Gaviota or Nalimov cache to maximum? Note: I restart the engines for each game.

(and if so, why? Does the entire cache need to be searched for stored results, even empty portions of it?)

TIA

You are addressing a very good point. The way memory is used make the comparisons difficult.

Gaviota TBs were not optimized to be that big (for the simple reason it would be better to have that memory used as hashtable and I never give more than 5% of HT) but it may be ok. I should look at this, since now humongous amount of memory are more available.

Miguel

[RJN]

Thanks for the reply, Miguel.

I have tried searching for an answer to the pros and cons of setting the EGTB cache "properly", but found little material, except what you pointed out, it could waste RAM. But with memory plentiful, I am curious what bad effects would arise from having the cache large. Does it only waste RAM, or does having a large cache noticeably lower the efficiency of the cache? (memory cache is still better than disk or even SSD access, I would think)

I have a 16GB system, so the largest hash I can use is 8GB, due to system overhead. This means I have plenty of RAM left for cache, yet most engines limit the cache to 1GB or less. Why this limit? I have done long analysis of endgames and seen the cache eventually fill, while wondering why it is limited to some seemingly arbitrary value. Why not a limit of, for example, 4GB?

I am curious what other authors have to say on this subject.

[bob]

Thanks for the reply, Miguel.

I have tried searching for an answer to the pros and cons of setting the EGTB cache "properly", but found little material, except what you pointed out, it could waste RAM. But with memory plentiful, I am curious what bad effects would arise from having the cache large. Does it only waste RAM, or does having a large cache noticeably lower the efficiency of the cache? (memory cache is still better than disk or even SSD access, I would think)

I have a 16GB system, so the largest hash I can use is 8GB, due to system overhead. This means I have plenty of RAM left for cache, yet most engines limit the cache to 1GB or less. Why this limit? I have done long analysis of endgames and seen the cache eventually fill, while wondering why it is limited to some seemingly arbitrary value. Why not a limit of, for example, 4GB?

I am curious what other authors have to say on this subject.

If you have lots of memory that is unused, it becomes cache ANYWAY. The O/S will cache as many blocks of files as it can. In the case of Nalimov, the only issue is that the filesystem caches compressed blocks, while the egtb cache holds uncompressed blocks, making those a bit quicker to access. But either way, I/O will be minimized if you have the memory, unless you eat it all up with hash table...

[RJN]

Thanks for the reply, Miguel.

I have tried searching for an answer to the pros and cons of setting the EGTB cache "properly", but found little material, except what you pointed out, it could waste RAM. But with memory plentiful, I am curious what bad effects would arise from having the cache large. Does it only waste RAM, or does having a large cache noticeably lower the efficiency of the cache? (memory cache is still better than disk or even SSD access, I would think)

I have a 16GB system, so the largest hash I can use is 8GB, due to system overhead. This means I have plenty of RAM left for cache, yet most engines limit the cache to 1GB or less. Why this limit? I have done long analysis of endgames and seen the cache eventually fill, while wondering why it is limited to some seemingly arbitrary value. Why not a limit of, for example, 4GB?

I am curious what other authors have to say on this subject.

If you have lots of memory that is unused, it becomes cache ANYWAY. The O/S will cache as many blocks of files as it can. In the case of Nalimov, the only issue is that the filesystem caches compressed blocks, while the egtb cache holds uncompressed blocks, making those a bit quicker to access. But either way, I/O will be minimized if you have the memory, unless you eat it all up with hash table...

Thanks, I take this to mean that on a 16 GB machine running a tournament with 512 MB hash, there is basically no downside to setting the TB cache to maximum of 1 GB.

I am only using Gaviota and Syzygy TBs in the tournament, no Nalimov. Syzygy, of course, has no cache parameter. I was thinking that setting the Gaviota cache high probably does not help too much during a 40 in 5 minute game, since the engines rarely uses more than 256MB Gaviota cache at that time control (even with low probe depths) but it would at least possibly limit Syzygy RAM availability a bit, making things fairer. Syzygy (especially SF Syzygy) gets MUCH higher TB hit counts than Gaviota, even with the probe depths for Gaviota set low. Also, I am using 6-piece Syzygy, which triggers earlier TB access.

For analysis though, I wish there was not a 1GB limit for Gaviota cache.

[mvk]

Setting the Gaviota cache very high might impact the Syzygy engines negatively. You need to reserve a healthy amount of OS cache for those.

[Mike S.]

The guideline I always recommend is:

Maximum is NOT optimum.

But I do not have data to back it up...

[RJN]

Setting the Gaviota cache very high might impact the Syzygy engines negatively. You need to reserve a healthy amount of OS cache for those.

I have more than 12GB physical free for Syzygy (OS) use during the tournament, so I don't think Syzygy would suffer too much +/- 1GB, no?

Probably an academic question thread, I was just wondering.

[mvk]

Setting the Gaviota cache very high might impact the Syzygy engines negatively. You need to reserve a healthy amount of OS cache for those.

I have more than 12GB physical free for Syzygy (OS) use during the tournament, so I don't think Syzygy would suffer too much +/- 1GB, no?

Probably an academic question thread, I was just wondering.

12GB (or 11GB) certainly seems sufficient. What also matters is if there is one or more games playing.

[ThomasJMiller]

a slightly offtopic quesiton: I'm thinking of buying a SSD for SyzygyBases. What specs should I take in consideration besides size of course?
I think IOPS value is important?