Page 1 of 2
Problem with SF6 and Syzygy TB
Posted: Wed Apr 01, 2015 2:47 am
by Zenmastur
Has anyone using Stockfish with Syzygy TB's seen a problem where working memory set increasing until all memory is used and then disk I/O goes thru the roof due to parts being swapped to disk or is it just me.
I'm using windows 7 with all current patches with minimal tasks in the back ground and 16Gb of ram. It doesn't take that long for SF's working memory set to use all available ram.
Regards,
Zen
Re: Problem with SF6 and Syzygy TB
Posted: Wed Apr 01, 2015 4:55 am
by APassionForCriminalJustic
Zenmastur wrote:Has anyone using Stockfish with Syzygy TB's seen a problem where working memory set increasing until all memory is used and then disk I/O goes thru the roof due to parts being swapped to disk or is it just me.
I'm using windows 7 with all current patches with minimal tasks in the back ground and 16Gb of ram. It doesn't take that long for SF's working memory set to use all available ram.
Regards,
Zen
The operating system should control the total amount of ram to be filled with the tablebases if I understood correctly seeing what Ronald wrote. Personally, my system doesn't go beyond 97 percent RAM usage. Your system should not be maxing the total ram out like that.
Re: Problem with SF6 and Syzygy TB
Posted: Wed Apr 01, 2015 5:23 am
by Ferdy
Zenmastur wrote:Has anyone using Stockfish with Syzygy TB's seen a problem where working memory set increasing until all memory is used and then disk I/O goes thru the roof due to parts being swapped to disk or is it just me.
I'm using windows 7 with all current patches with minimal tasks in the back ground and 16Gb of ram. It doesn't take that long for SF's working memory set to use all available ram.
Regards,
Zen
Analyzing ending position using 1 thread and 5-men sy, I got private bytes used increasing at the rate of 327 bytes/sec on around 18 minutes of analysis session. This is on win7 64 bit.
Re: Problem with SF6 and Syzygy TB
Posted: Wed Apr 01, 2015 7:08 am
by Modern Times
APassionForCriminalJustic wrote:
The operating system should control the total amount of ram to be filled with the tablebases if I understood correctly seeing what Ronald wrote. Personally, my system doesn't go beyond 97 percent RAM usage. Your system should not be maxing the total ram out like that.
Same here, never a problem. I'm on Windows 8.1 and it handles it perfectly. Maybe Windows 7 doesn't do so well. It gets to 97% but will immediately release RAM if you want to open another application for example.
Re: Problem with SF6 and Syzygy TB
Posted: Wed Apr 01, 2015 9:08 am
by syzygy
Zenmastur wrote:I'm using windows 7 with all current patches with minimal tasks in the back ground and 16Gb of ram. It doesn't take that long for SF's working memory set to use all available ram.
If you use 8GB of hash tables, it might help to reduce that to 4GB. Alternatively, or in addition, increasing probe depth should help as well.
Ideally, the OS should not swap memory to disk when under memory pressure but instead release RAM used for caching TB data. But I don't know if all OS's are so smart. (I'm not even sure what Linux does in this case.)
Re: Problem with SF6 and Syzygy TB
Posted: Wed Apr 01, 2015 9:10 am
by syzygy
Modern Times wrote:Same here, never a problem. I'm on Windows 8.1 and it handles it perfectly. Maybe Windows 7 doesn't do so well. It gets to 97% but will immediately release RAM if you want to open another application for example.
Even if the engine process continues to run (or at least exist)? That would be the ideal behaviour.
Re: Problem with SF6 and Syzygy TB
Posted: Wed Apr 01, 2015 9:18 am
by Zenmastur
Modern Times wrote:APassionForCriminalJustic wrote:
The operating system should control the total amount of ram to be filled with the tablebases if I understood correctly seeing what Ronald wrote. Personally, my system doesn't go beyond 97 percent RAM usage. Your system should not be maxing the total ram out like that.
Same here, never a problem. I'm on Windows 8.1 and it handles it perfectly. Maybe Windows 7 doesn't do so well. It gets to 97% but will immediately release RAM if you want to open another application for example.
Hmmm...
Most of the time it does that on my system as well...
It seems like there is a problem when Scid vs PC wants to write something to disk. i.e. ad told it to add the current PV as a variation and save the game. The swap file is on a SSD and is different than the drive the Scid is writing to. Just the same, the swap file reads and writes go thru the roof and the system grinds to halt for 20 minutes or more while performing a simple update of the game file.
I noticed this problem before and stopped using tablebases for this reason, but never investigated why and had forgotten about it. Its really annoying when it happens as the system becomes useless until the disk I/O's die down which can take a while.
Is there some way to limit the amount of ram syzygy TB's use?
Regards,
Zen
Re: Problem with SF6 and Syzygy TB
Posted: Wed Apr 01, 2015 9:35 am
by Michel
Zenmastur wrote:Modern Times wrote:APassionForCriminalJustic wrote:
It seems like there is a problem when Scid vs PC wants to write something to disk. i.e. ad told it to add the current PV as a variation and save the game. The swap file is on a SSD and is different than the drive the Scid is writing to. Just the same, the swap file reads and writes go thru the roof and the system grinds to halt for 20 minutes or more while performing a simple update of the game file.
Zen
Note that the use of syzygy should not cause the use of the swapfile since the syzygy tbs are _already_ on disk.
The swap file is for heap allocated memory.
Re: Problem with SF6 and Syzygy TB
Posted: Wed Apr 01, 2015 9:47 am
by Zenmastur
Michel wrote:Zenmastur wrote:Modern Times wrote:APassionForCriminalJustic wrote:
It seems like there is a problem when Scid vs PC wants to write something to disk. i.e. ad told it to add the current PV as a variation and save the game. The swap file is on a SSD and is different than the drive the Scid is writing to. Just the same, the swap file reads and writes go thru the roof and the system grinds to halt for 20 minutes or more while performing a simple update of the game file.
Zen
Note that the use of syzygy should not cause the use of the swapfile since the syzygy tbs are _already_ on disk.
The swap file is for heap allocated memory.
Well, I looked at the resource monitor while this was happening and it clearly showed SF6 as causing a large number of faults, the swap file became very active, I can tell because its on a separate drive. Scid, SF, and Resource monitor were the only apps running.
I don't know what would cause it, but this isn't the first time I've had this problem with SF/Syzygy.
Regards,
Zen
Re: Problem with SF6 and Syzygy TB
Posted: Wed Apr 01, 2015 10:02 pm
by Zenmastur
syzygy wrote:Zenmastur wrote:I'm using windows 7 with all current patches with minimal tasks in the back ground and 16Gb of ram. It doesn't take that long for SF's working memory set to use all available ram.
If you use 8GB of hash tables, it might help to reduce that to 4GB. Alternatively, or in addition, increasing probe depth should help as well.
Ideally, the OS should not swap memory to disk when under memory pressure but instead release RAM used for caching TB data. But I don't know if all OS's are so smart. (I'm not even sure what Linux does in this case.)
Most of the time I use 4 GB of cache. But even if I reduce this to 1 GB It will still run ram use-age to greater than 16,000 MB. Probe Depth is set at default, which is six if I recall properly. I'm wondering why it shows SF as using 16,000+ MB in the resource console if the OS is doing the caching. Is it because the cache is directly mapped by the program?
I know there is a way to remove memory from the working pool so that it can't be swapped out by the OS. This is particularly useful for things like RAM disks and some programs that do their own caching. Unfortunately this can't be used with SF and their doesn't appear to be any way to curb Syzygy's use of the system cache, so for now I guess I'll try using only 5-men table-bases and see if this helps.
Regards,
Zen