7-men Syzygy attempt
Moderator: Ras
-
- Posts: 5665
- Joined: Tue Feb 28, 2012 11:56 pm
Re: 7-men Syzygy attempt
To prevent your programs from being swapped out, disable the page file. Windows will then release RAM caching TB data when it needs more memory.
-
- Posts: 2821
- Joined: Fri Sep 25, 2015 9:38 pm
- Location: Sortland, Norway
Re: 7-men Syzygy attempt
Is it possible to set a quota. Instead of oscillating between 97-98% of system RAM, what about setting 10% for OS. Now syzygy get cached up to 90%. The remaining 10% is for system.syzygy wrote:To prevent your programs from being swapped out, disable the page file. Windows will then release RAM caching TB data when it needs more memory.
The idea here is improved performance, e.g. background task is unlikely to interfer. 10% safety margin. This can be be available as option.
-
- Posts: 694
- Joined: Sun Nov 08, 2015 11:10 pm
- Full name: Bojun Guo
Re: 7-men Syzygy attempt
In the cases where you don't think the OS can do a better job managing caches, you claim a piece of memory and maintain it on your own by implementing something like TB cache.
Then to the OS's view it is just a piece of memory and it will still do file caching when it thinks it is necessary.
You will probably end up with less memory space on both caches and usually it turns out worse, unless of course in cases where TB cache stores decompressed data and it can outperform the decompression speed, which is likely not the case with Syzygy.
As for the quota, we use the cache in a way that the OS can release it with almost zero overhead, because the contents are not changed in any form that requires a file sync. So it is totally fine with the OS dropping file caches from 99.99% of memory to 0% if there is something else that needed the memory space. But the other way around then the system can't just free your own cache in order to claim more free memory and thus would fall back to swapping and that usually hangs everything.
Then to the OS's view it is just a piece of memory and it will still do file caching when it thinks it is necessary.
You will probably end up with less memory space on both caches and usually it turns out worse, unless of course in cases where TB cache stores decompressed data and it can outperform the decompression speed, which is likely not the case with Syzygy.
As for the quota, we use the cache in a way that the OS can release it with almost zero overhead, because the contents are not changed in any form that requires a file sync. So it is totally fine with the OS dropping file caches from 99.99% of memory to 0% if there is something else that needed the memory space. But the other way around then the system can't just free your own cache in order to claim more free memory and thus would fall back to swapping and that usually hangs everything.
-
- Posts: 2821
- Joined: Fri Sep 25, 2015 9:38 pm
- Location: Sortland, Norway
Re: 7-men Syzygy attempt
Page file is already disabled.noobpwnftw wrote:As for the quota, we use the cache in a way that the OS can release it with almost zero overhead, because the contents are not changed in any form that requires a file sync. So it is totally fine with the OS dropping file caches from 99.99% of memory to 0% if there is something else that needed the memory space. But the other way around then the system can't just free your own cache in order to claim more free memory and thus would fall back to swapping and that usually hangs everything.
Here is my concept visualized:

Blueprint concept, internal cache depends on total system ram. 32 GB system can afford to give syzygy cache more portion.
-
- Posts: 694
- Joined: Sun Nov 08, 2015 11:10 pm
- Full name: Bojun Guo
Re: 7-men Syzygy attempt
Actually what happens as it is though, is that system will give Syzygy as much as it can until something else is claiming that cache memory.Nordlandia wrote: Blueprint concept, internal cache depends on total system ram. 32 GB system can afford to give syzygy cache more portion.
To prove this you can do an "echo *.* > NUL" while watching the task manager on Windows to see that a big portion of free memory will be claimed as buffer/cache or whatever it is called.
Same thing in Linux if you do "cat *.* > /dev/null"
-
- Posts: 2821
- Joined: Fri Sep 25, 2015 9:38 pm
- Location: Sortland, Norway
Re: 7-men Syzygy attempt
Bojun Guo: as long other tasks make claim on memory get higher priority than syzygy.
-
- Posts: 5665
- Joined: Tue Feb 28, 2012 11:56 pm
Re: 7-men Syzygy attempt
If you have disabled the page file, your system is not swapping to disk anymore. I'm not sure what remaining problem there could be? (That your conceptual idea of 50% unused=useless RAM is not realised is not a problem that I want to solve.)
-
- Posts: 2821
- Joined: Fri Sep 25, 2015 9:38 pm
- Location: Sortland, Norway
Re: 7-men Syzygy attempt
Everything is fine. I was just slightly worried switching back and forth 97-98% is inferior for performance. He said other tasks make claim on memory get higher priority than syzygy, so OS has to toss out that space for that particular task. If i'm not mistaken.
-
- Posts: 694
- Joined: Sun Nov 08, 2015 11:10 pm
- Full name: Bojun Guo
Re: 7-men Syzygy attempt
Correct, anything that actually claims memory(background or not) gets priority over those OS caches and the OS usually picks the least useful caches to drop. So no, switching back and forth on cache usage in general does virtually no performance harm in regard to what has to be done to get everything going, if you don't want anything else gets in the way of those caches then simply don't run it.Nordlandia wrote:Everything is fine. I was just slightly worried switching back and forth 97-98% is inferior for performance. He said other tasks make claim on memory get higher priority than syzygy, so OS has to toss out that space for that particular task. If i'm not mistaken.

-
- Posts: 28274
- Joined: Fri Mar 10, 2006 10:06 am
- Location: Amsterdam
- Full name: H G Muller
Re: 7-men Syzygy attempt
If it is not allowed to use a page file, there should not be any application memory the OS can toss out. This guarantees continuous presence of all tasks in memoy, and limits the amount of RAM that can be used for disk caching to that which would otherwise remain idle. This has nothing to do with priority. Even the lowest priority task must remain in memory, if it is not terminated. Because there is no other place it could be put instead.Nordlandia wrote:Everything is fine. I was just slightly worried switching back and forth 97-98% is inferior for performance. He said other tasks make claim on memory get higher priority than syzygy, so OS has to toss out that space for that particular task. If i'm not mistaken.