TalkChess.com

To prevent your programs from being swapped out, disable the page file. Windows will then release RAM caching TB data when it needs more memory.

syzygy wrote:To prevent your programs from being swapped out, disable the page file. Windows will then release RAM caching TB data when it needs more memory.

Is it possible to set a quota. Instead of oscillating between 97-98% of system RAM, what about setting 10% for OS. Now syzygy get cached up to 90%. The remaining 10% is for system.

The idea here is improved performance, e.g. background task is unlikely to interfer. 10% safety margin. This can be be available as option.

In the cases where you don't think the OS can do a better job managing caches, you claim a piece of memory and maintain it on your own by implementing something like TB cache.
Then to the OS's view it is just a piece of memory and it will still do file caching when it thinks it is necessary.

You will probably end up with less memory space on both caches and usually it turns out worse, unless of course in cases where TB cache stores decompressed data and it can outperform the decompression speed, which is likely not the case with Syzygy.

As for the quota, we use the cache in a way that the OS can release it with almost zero overhead, because the contents are not changed in any form that requires a file sync. So it is totally fine with the OS dropping file caches from 99.99% of memory to 0% if there is something else that needed the memory space. But the other way around then the system can't just free your own cache in order to claim more free memory and thus would fall back to swapping and that usually hangs everything.

noobpwnftw wrote:As for the quota, we use the cache in a way that the OS can release it with almost zero overhead, because the contents are not changed in any form that requires a file sync. So it is totally fine with the OS dropping file caches from 99.99% of memory to 0% if there is something else that needed the memory space. But the other way around then the system can't just free your own cache in order to claim more free memory and thus would fall back to swapping and that usually hangs everything.

Page file is already disabled.

Here is my concept visualized:

Blueprint concept, internal cache depends on total system ram. 32 GB system can afford to give syzygy cache more portion.

Nordlandia wrote: Blueprint concept, internal cache depends on total system ram. 32 GB system can afford to give syzygy cache more portion.

Actually what happens as it is though, is that system will give Syzygy as much as it can until something else is claiming that cache memory.
To prove this you can do an "echo *.* > NUL" while watching the task manager on Windows to see that a big portion of free memory will be claimed as buffer/cache or whatever it is called.
Same thing in Linux if you do "cat *.* > /dev/null"

Bojun Guo: as long other tasks make claim on memory get higher priority than syzygy.

If you have disabled the page file, your system is not swapping to disk anymore. I'm not sure what remaining problem there could be? (That your conceptual idea of 50% unused=useless RAM is not realised is not a problem that I want to solve.)

Everything is fine. I was just slightly worried switching back and forth 97-98% is inferior for performance. He said other tasks make claim on memory get higher priority than syzygy, so OS has to toss out that space for that particular task. If i'm not mistaken.

Nordlandia wrote:Everything is fine. I was just slightly worried switching back and forth 97-98% is inferior for performance. He said other tasks make claim on memory get higher priority than syzygy, so OS has to toss out that space for that particular task. If i'm not mistaken.

Correct, anything that actually claims memory(background or not) gets priority over those OS caches and the OS usually picks the least useful caches to drop. So no, switching back and forth on cache usage in general does virtually no performance harm in regard to what has to be done to get everything going, if you don't want anything else gets in the way of those caches then simply don't run it.

Nordlandia wrote:Everything is fine. I was just slightly worried switching back and forth 97-98% is inferior for performance. He said other tasks make claim on memory get higher priority than syzygy, so OS has to toss out that space for that particular task. If i'm not mistaken.

If it is not allowed to use a page file, there should not be any application memory the OS can toss out. This guarantees continuous presence of all tasks in memoy, and limits the amount of RAM that can be used for disk caching to that which would otherwise remain idle. This has nothing to do with priority. Even the lowest priority task must remain in memory, if it is not terminated. Because there is no other place it could be put instead.