Problem with SF6 and Syzygy TB

Discussion of chess software programming and technical issues.

Moderators: hgm, Harvey Williamson, bob

Zenmastur
Posts: 272
Joined: Sat May 31, 2014 6:28 am

Problem with SF6 and Syzygy TB

Post by Zenmastur » Wed Apr 01, 2015 12:47 am

Has anyone using Stockfish with Syzygy TB's seen a problem where working memory set increasing until all memory is used and then disk I/O goes thru the roof due to parts being swapped to disk or is it just me.

I'm using windows 7 with all current patches with minimal tasks in the back ground and 16Gb of ram. It doesn't take that long for SF's working memory set to use all available ram.

Regards,

Zen
Only 2 defining forces have ever offered to die for you.....Jesus Christ and the American Soldier. One died for your soul, the other for your freedom.

APassionForCriminalJustic
Posts: 415
Joined: Sat May 24, 2014 7:16 am

Re: Problem with SF6 and Syzygy TB

Post by APassionForCriminalJustic » Wed Apr 01, 2015 2:55 am

Zenmastur wrote:Has anyone using Stockfish with Syzygy TB's seen a problem where working memory set increasing until all memory is used and then disk I/O goes thru the roof due to parts being swapped to disk or is it just me.

I'm using windows 7 with all current patches with minimal tasks in the back ground and 16Gb of ram. It doesn't take that long for SF's working memory set to use all available ram.

Regards,

Zen
The operating system should control the total amount of ram to be filled with the tablebases if I understood correctly seeing what Ronald wrote. Personally, my system doesn't go beyond 97 percent RAM usage. Your system should not be maxing the total ram out like that.

Ferdy
Posts: 3597
Joined: Sun Aug 10, 2008 1:15 pm
Location: Philippines

Re: Problem with SF6 and Syzygy TB

Post by Ferdy » Wed Apr 01, 2015 3:23 am

Zenmastur wrote:Has anyone using Stockfish with Syzygy TB's seen a problem where working memory set increasing until all memory is used and then disk I/O goes thru the roof due to parts being swapped to disk or is it just me.

I'm using windows 7 with all current patches with minimal tasks in the back ground and 16Gb of ram. It doesn't take that long for SF's working memory set to use all available ram.

Regards,

Zen
Analyzing ending position using 1 thread and 5-men sy, I got private bytes used increasing at the rate of 327 bytes/sec on around 18 minutes of analysis session. This is on win7 64 bit.

Modern Times
Posts: 2141
Joined: Thu Jun 07, 2012 9:02 pm

Re: Problem with SF6 and Syzygy TB

Post by Modern Times » Wed Apr 01, 2015 5:08 am

APassionForCriminalJustic wrote:
The operating system should control the total amount of ram to be filled with the tablebases if I understood correctly seeing what Ronald wrote. Personally, my system doesn't go beyond 97 percent RAM usage. Your system should not be maxing the total ram out like that.
Same here, never a problem. I'm on Windows 8.1 and it handles it perfectly. Maybe Windows 7 doesn't do so well. It gets to 97% but will immediately release RAM if you want to open another application for example.

syzygy
Posts: 4221
Joined: Tue Feb 28, 2012 10:56 pm

Re: Problem with SF6 and Syzygy TB

Post by syzygy » Wed Apr 01, 2015 7:08 am

Zenmastur wrote:I'm using windows 7 with all current patches with minimal tasks in the back ground and 16Gb of ram. It doesn't take that long for SF's working memory set to use all available ram.
If you use 8GB of hash tables, it might help to reduce that to 4GB. Alternatively, or in addition, increasing probe depth should help as well.

Ideally, the OS should not swap memory to disk when under memory pressure but instead release RAM used for caching TB data. But I don't know if all OS's are so smart. (I'm not even sure what Linux does in this case.)

syzygy
Posts: 4221
Joined: Tue Feb 28, 2012 10:56 pm

Re: Problem with SF6 and Syzygy TB

Post by syzygy » Wed Apr 01, 2015 7:10 am

Modern Times wrote:Same here, never a problem. I'm on Windows 8.1 and it handles it perfectly. Maybe Windows 7 doesn't do so well. It gets to 97% but will immediately release RAM if you want to open another application for example.
Even if the engine process continues to run (or at least exist)? That would be the ideal behaviour.

Zenmastur
Posts: 272
Joined: Sat May 31, 2014 6:28 am

Re: Problem with SF6 and Syzygy TB

Post by Zenmastur » Wed Apr 01, 2015 7:18 am

Modern Times wrote:
APassionForCriminalJustic wrote:
The operating system should control the total amount of ram to be filled with the tablebases if I understood correctly seeing what Ronald wrote. Personally, my system doesn't go beyond 97 percent RAM usage. Your system should not be maxing the total ram out like that.
Same here, never a problem. I'm on Windows 8.1 and it handles it perfectly. Maybe Windows 7 doesn't do so well. It gets to 97% but will immediately release RAM if you want to open another application for example.
Hmmm...

Most of the time it does that on my system as well...

It seems like there is a problem when Scid vs PC wants to write something to disk. i.e. ad told it to add the current PV as a variation and save the game. The swap file is on a SSD and is different than the drive the Scid is writing to. Just the same, the swap file reads and writes go thru the roof and the system grinds to halt for 20 minutes or more while performing a simple update of the game file.

I noticed this problem before and stopped using tablebases for this reason, but never investigated why and had forgotten about it. Its really annoying when it happens as the system becomes useless until the disk I/O's die down which can take a while.

Is there some way to limit the amount of ram syzygy TB's use?

Regards,

Zen
Only 2 defining forces have ever offered to die for you.....Jesus Christ and the American Soldier. One died for your soul, the other for your freedom.

Michel
Posts: 1960
Joined: Sun Sep 28, 2008 11:50 pm

Re: Problem with SF6 and Syzygy TB

Post by Michel » Wed Apr 01, 2015 7:35 am

Zenmastur wrote:
Modern Times wrote:
APassionForCriminalJustic wrote:
It seems like there is a problem when Scid vs PC wants to write something to disk. i.e. ad told it to add the current PV as a variation and save the game. The swap file is on a SSD and is different than the drive the Scid is writing to. Just the same, the swap file reads and writes go thru the roof and the system grinds to halt for 20 minutes or more while performing a simple update of the game file.


Zen
Note that the use of syzygy should not cause the use of the swapfile since the syzygy tbs are _already_ on disk.

The swap file is for heap allocated memory.
Ideas=science. Simplification=engineering.
Without ideas there is nothing to simplify.

Zenmastur
Posts: 272
Joined: Sat May 31, 2014 6:28 am

Re: Problem with SF6 and Syzygy TB

Post by Zenmastur » Wed Apr 01, 2015 7:47 am

Michel wrote:
Zenmastur wrote:
Modern Times wrote:
APassionForCriminalJustic wrote:
It seems like there is a problem when Scid vs PC wants to write something to disk. i.e. ad told it to add the current PV as a variation and save the game. The swap file is on a SSD and is different than the drive the Scid is writing to. Just the same, the swap file reads and writes go thru the roof and the system grinds to halt for 20 minutes or more while performing a simple update of the game file.


Zen
Note that the use of syzygy should not cause the use of the swapfile since the syzygy tbs are _already_ on disk.

The swap file is for heap allocated memory.

Well, I looked at the resource monitor while this was happening and it clearly showed SF6 as causing a large number of faults, the swap file became very active, I can tell because its on a separate drive. Scid, SF, and Resource monitor were the only apps running.

I don't know what would cause it, but this isn't the first time I've had this problem with SF/Syzygy.

Regards,

Zen
Only 2 defining forces have ever offered to die for you.....Jesus Christ and the American Soldier. One died for your soul, the other for your freedom.

Zenmastur
Posts: 272
Joined: Sat May 31, 2014 6:28 am

Re: Problem with SF6 and Syzygy TB

Post by Zenmastur » Wed Apr 01, 2015 8:02 pm

syzygy wrote:
Zenmastur wrote:I'm using windows 7 with all current patches with minimal tasks in the back ground and 16Gb of ram. It doesn't take that long for SF's working memory set to use all available ram.
If you use 8GB of hash tables, it might help to reduce that to 4GB. Alternatively, or in addition, increasing probe depth should help as well.

Ideally, the OS should not swap memory to disk when under memory pressure but instead release RAM used for caching TB data. But I don't know if all OS's are so smart. (I'm not even sure what Linux does in this case.)
Most of the time I use 4 GB of cache. But even if I reduce this to 1 GB It will still run ram use-age to greater than 16,000 MB. Probe Depth is set at default, which is six if I recall properly. I'm wondering why it shows SF as using 16,000+ MB in the resource console if the OS is doing the caching. Is it because the cache is directly mapped by the program?

I know there is a way to remove memory from the working pool so that it can't be swapped out by the OS. This is particularly useful for things like RAM disks and some programs that do their own caching. Unfortunately this can't be used with SF and their doesn't appear to be any way to curb Syzygy's use of the system cache, so for now I guess I'll try using only 5-men table-bases and see if this helps.

Regards,

Zen
Only 2 defining forces have ever offered to die for you.....Jesus Christ and the American Soldier. One died for your soul, the other for your freedom.

Post Reply