7-men Syzygy attempt

Discussion of chess software programming and technical issues.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
Post Reply
syzygy
Posts: 4458
Joined: Tue Feb 28, 2012 10:56 pm

Re: 7-men Syzygy attempt

Post by syzygy » Mon Mar 19, 2018 11:46 am

To prevent your programs from being swapped out, disable the page file. Windows will then release RAM caching TB data when it needs more memory.

User avatar
Nordlandia
Posts: 2471
Joined: Fri Sep 25, 2015 7:38 pm
Location: Sortland, Norway

Re: 7-men Syzygy attempt

Post by Nordlandia » Mon Mar 19, 2018 1:53 pm

syzygy wrote:To prevent your programs from being swapped out, disable the page file. Windows will then release RAM caching TB data when it needs more memory.
Is it possible to set a quota. Instead of oscillating between 97-98% of system RAM, what about setting 10% for OS. Now syzygy get cached up to 90%. The remaining 10% is for system.

The idea here is improved performance, e.g. background task is unlikely to interfer. 10% safety margin. This can be be available as option.

noobpwnftw
Posts: 360
Joined: Sun Nov 08, 2015 10:10 pm

Re: 7-men Syzygy attempt

Post by noobpwnftw » Mon Mar 19, 2018 2:19 pm

In the cases where you don't think the OS can do a better job managing caches, you claim a piece of memory and maintain it on your own by implementing something like TB cache.
Then to the OS's view it is just a piece of memory and it will still do file caching when it thinks it is necessary.

You will probably end up with less memory space on both caches and usually it turns out worse, unless of course in cases where TB cache stores decompressed data and it can outperform the decompression speed, which is likely not the case with Syzygy.

As for the quota, we use the cache in a way that the OS can release it with almost zero overhead, because the contents are not changed in any form that requires a file sync. So it is totally fine with the OS dropping file caches from 99.99% of memory to 0% if there is something else that needed the memory space. But the other way around then the system can't just free your own cache in order to claim more free memory and thus would fall back to swapping and that usually hangs everything.

User avatar
Nordlandia
Posts: 2471
Joined: Fri Sep 25, 2015 7:38 pm
Location: Sortland, Norway

Re: 7-men Syzygy attempt

Post by Nordlandia » Mon Mar 19, 2018 2:47 pm

noobpwnftw wrote:As for the quota, we use the cache in a way that the OS can release it with almost zero overhead, because the contents are not changed in any form that requires a file sync. So it is totally fine with the OS dropping file caches from 99.99% of memory to 0% if there is something else that needed the memory space. But the other way around then the system can't just free your own cache in order to claim more free memory and thus would fall back to swapping and that usually hangs everything.
Page file is already disabled.

Here is my concept visualized: Image

Blueprint concept, internal cache depends on total system ram. 32 GB system can afford to give syzygy cache more portion.

noobpwnftw
Posts: 360
Joined: Sun Nov 08, 2015 10:10 pm

Re: 7-men Syzygy attempt

Post by noobpwnftw » Mon Mar 19, 2018 3:04 pm

Nordlandia wrote: Blueprint concept, internal cache depends on total system ram. 32 GB system can afford to give syzygy cache more portion.
Actually what happens as it is though, is that system will give Syzygy as much as it can until something else is claiming that cache memory.
To prove this you can do an "echo *.* > NUL" while watching the task manager on Windows to see that a big portion of free memory will be claimed as buffer/cache or whatever it is called.
Same thing in Linux if you do "cat *.* > /dev/null"

User avatar
Nordlandia
Posts: 2471
Joined: Fri Sep 25, 2015 7:38 pm
Location: Sortland, Norway

Re: 7-men Syzygy attempt

Post by Nordlandia » Mon Mar 19, 2018 3:58 pm

Bojun Guo: as long other tasks make claim on memory get higher priority than syzygy.

syzygy
Posts: 4458
Joined: Tue Feb 28, 2012 10:56 pm

Re: 7-men Syzygy attempt

Post by syzygy » Mon Mar 19, 2018 4:57 pm

If you have disabled the page file, your system is not swapping to disk anymore. I'm not sure what remaining problem there could be? (That your conceptual idea of 50% unused=useless RAM is not realised is not a problem that I want to solve.)

User avatar
Nordlandia
Posts: 2471
Joined: Fri Sep 25, 2015 7:38 pm
Location: Sortland, Norway

Re: 7-men Syzygy attempt

Post by Nordlandia » Mon Mar 19, 2018 5:38 pm

Everything is fine. I was just slightly worried switching back and forth 97-98% is inferior for performance. He said other tasks make claim on memory get higher priority than syzygy, so OS has to toss out that space for that particular task. If i'm not mistaken.

noobpwnftw
Posts: 360
Joined: Sun Nov 08, 2015 10:10 pm

Re: 7-men Syzygy attempt

Post by noobpwnftw » Mon Mar 19, 2018 7:06 pm

Nordlandia wrote:Everything is fine. I was just slightly worried switching back and forth 97-98% is inferior for performance. He said other tasks make claim on memory get higher priority than syzygy, so OS has to toss out that space for that particular task. If i'm not mistaken.
Correct, anything that actually claims memory(background or not) gets priority over those OS caches and the OS usually picks the least useful caches to drop. So no, switching back and forth on cache usage in general does virtually no performance harm in regard to what has to be done to get everything going, if you don't want anything else gets in the way of those caches then simply don't run it. :)

User avatar
hgm
Posts: 23772
Joined: Fri Mar 10, 2006 9:06 am
Location: Amsterdam
Full name: H G Muller
Contact:

Re: 7-men Syzygy attempt

Post by hgm » Mon Mar 19, 2018 7:25 pm

Nordlandia wrote:Everything is fine. I was just slightly worried switching back and forth 97-98% is inferior for performance. He said other tasks make claim on memory get higher priority than syzygy, so OS has to toss out that space for that particular task. If i'm not mistaken.
If it is not allowed to use a page file, there should not be any application memory the OS can toss out. This guarantees continuous presence of all tasks in memoy, and limits the amount of RAM that can be used for disk caching to that which would otherwise remain idle. This has nothing to do with priority. Even the lowest priority task must remain in memory, if it is not terminated. Because there is no other place it could be put instead.

Post Reply