Zenmastur wrote:syzygy wrote:Yes, the memory pages containing cached TB data show up as part of the process's memory in the Working Set column of the task manager. The Commit size column will look even worse, but that's just a number.
You're probably not doing this, but just in case, don't store TB files in a RAM disk. That will only make them use RAM twice: once in the RAM disk, once in the page cache. (A good RAM disk might avoid this problem but I doubt they exist for Windows.)
I don't have the TB file on a RAM disk at least for now due to memory limitations, they're on a SSD. I do have and use a ram disk, but I don't know if the version I have removes the memory it uses from the working pool. Even if it could, I haven't set up the OS to allow this, although it's not difficult to do.
Let me put it like this: RAM disks are simply a bad idea.
If the OS wants to use the data stored in the RAM disk, that data has to be copied to other parts of RAM (the page cache). So everything will simply take up memory _twice_. That is almost never a good idea. It is a very bad idea if you are trying to cope with 68GB of TB data on a 16GB system.
syzygy wrote:Just thought of something.
If you have 16GB and the WDL tables on SSD, simply disable your swap file. (I think it's called page file on Windows.)
Without a swap file, the system will not swap. Problem solved. When the system runs out of memory (because too much memory is used for caching TB data), the OS will have no other choice than to release memory pages caching TB data. And that is just what we want.
(I'm not kidding or trying to be sarcastic. I really doubt that for normal users, including heavy chess engine users, there is any practical advantage in having a swap file on a 16GB system.)
I would try this but, I do need more than 16 GB as I often use more than this, sometimes a lot more. It's surprisingly easy to do. I guess that makes me a non-normal user. My next system, coming soon I hope, will be a little more robust in the ram and mass storage departments.
You often need more than 16 GB, but still use a RAM disk? I'm afraid there is no logic in that.
In the time of 16 MB computers having another 16 MB of swap was a very good idea. The system would at some point run out of memory, and swapping a few MB of RAM to the pagefile was a relatively fast operation. Better to wait maybe a second than to get an out of memory error and a program that quits.
But now imagine a 16 GB system with another 16 GB of swap. Swapping a few GB to the pagefile will freeze your computer for many minutes. That makes your system unusable. Of course if you _really_ need 20 GB and only have 16 GB of physical RAM and lots of patience, then a pagefile will eventually do the trick, but this exceptional and far from ideal. In general, it is better to get an out of memory error if the 16 GB are exceeded, because most likely something went quite wrong and the alternative is a system that is unresponsive for many many minutes after which the user loses hope and forces a reboot.
Disabling any RAM disks and the page file will most likely make any modern system much more responsive.
I'm going to try a few things, if they don't help, I'll probably just go back to not using TB's for most tasks. They're nice in some instances, but not mandatory for most.
5-piece TBs won't eat more than 400MB.
