I assume Windows caches all files that get read from disk (SSD or HDD) without prejudice. Is there a way to prevent certain file types from being cached? I would think doing so when a DTZ is read is a waste of memory cache capacity. The largest DTZ files are over 2GB.
During a tournament, this wasted cached would then carry over when the next game's WDL are read, with some of the last previous game's WDL kicked out of RAM by the previous game DTZ read once 6 (or 5) pieces had been reached.
If they are cached, what possible way is there to keep *.rtbz from being stored in RAM when they get read off of storage?
Then again, maybe it's better that the DTZ was cached in the previous game while that was being played, so it does not have to be read over and over, despite the fact it should only take a few seconds if DTZ is in a SSD.
Or does Syzygy Bases already take this point into account in some manner?
TIA
Syzygy DTZ caching in Windows. Waste of memory?
Moderators: hgm, Rebel, chrisw
-
- Posts: 303
- Joined: Fri Jun 21, 2013 5:18 am
- Location: Orion Spiral Arm
-
- Posts: 5563
- Joined: Tue Feb 28, 2012 11:56 pm
Re: Syzygy DTZ caching in Windows. Waste of memory?
They are only accessed at the end of the game, when the 6-piece position is already on the board and the game is effectively over. And then only small parts are accessed and cached. How small those parts is decided by your OS, but an OS is supposed to make reasonable decisions in this respect.RJN wrote:I assume Windows caches all files that get read from disk (SSD or HDD) without prejudice. Is there a way to prevent certain file types from being cached? I would think doing so when a DTZ is read is a waste of memory cache capacity. The largest DTZ files are over 2GB.
So there is no problem, I would say.
-
- Posts: 303
- Joined: Fri Jun 21, 2013 5:18 am
- Location: Orion Spiral Arm
Re: Syzygy DTZ caching in Windows. Waste of memory?
Thanks. I knew they were only accessed at the end of a game. That's why I was wondering about the effect of the next game in a tournament, when the DTZ might have overwritten some WDL cache that is needed in succeeding games.
I guess the fact that only small parts are accessed at a time means it is a non-issue. I had assumed the entire file was read when a DTZ condition happened.
I guess the fact that only small parts are accessed at a time means it is a non-issue. I had assumed the entire file was read when a DTZ condition happened.
-
- Posts: 5563
- Joined: Tue Feb 28, 2012 11:56 pm
Re: Syzygy DTZ caching in Windows. Waste of memory?
If a small part of the WDL cache had to be overwritten, then that just had to happen. The DTZ data did have to be read.RJN wrote:Thanks. I knew they were only accessed at the end of a game. That's why I was wondering about the effect of the next game in a tournament, when the DTZ might have overwritten some WDL cache that is needed in succeeding games.
Then accessing DTZ files would be really slow (loading 2 GB just to access a few bytes). Memory mapped files are loaded using "demand paging":I guess the fact that only small parts are accessed at a time means it is a non-issue. I had assumed the entire file was read when a DTZ condition happened.
http://en.wikipedia.org/wiki/Demand_paging
To understand the third sentence of the initial paragraph you have to know that when a program is launched, its code is not completely loaded from disk into RAM, but memory-mapped into RAM. The relevant code pages are then paged into RAM as the program executes. The same happens with TB files, which are fully memory-mapped into memory on first access, but not fully read from disk.
Memory-mapped data that is loaded into RAM may still be freed again when RAM is needed for other purposes.
-
- Posts: 303
- Joined: Fri Jun 21, 2013 5:18 am
- Location: Orion Spiral Arm
Re: Syzygy DTZ caching in Windows. Waste of memory?
I'll trust you on that all that, I'm not the best codersyzygy wrote:If a small part of the WDL cache had to be overwritten, then that just had to happen. The DTZ data did have to be read.RJN wrote:Thanks. I knew they were only accessed at the end of a game. That's why I was wondering about the effect of the next game in a tournament, when the DTZ might have overwritten some WDL cache that is needed in succeeding games.
Then accessing DTZ files would be really slow (loading 2 GB just to access a few bytes). Memory mapped files are loaded using "demand paging":I guess the fact that only small parts are accessed at a time means it is a non-issue. I had assumed the entire file was read when a DTZ condition happened.
http://en.wikipedia.org/wiki/Demand_paging
To understand the third sentence of the initial paragraph you have to know that when a program is launched, its code is not completely loaded from disk into RAM, but memory-mapped into RAM. The relevant code pages are then paged into RAM as the program executes. The same happens with TB files, which are fully memory-mapped into memory on first access, but not fully read from disk.
Memory-mapped data that is loaded into RAM may still be freed again when RAM is needed for other purposes.
Again, thanks for your amazing TBs. An impressive achievement that has become the gold standard of endgame performance. One of the reasons my new machine has 64GB DDR4 are to cache Syzygy bases.
-
- Posts: 5563
- Joined: Tue Feb 28, 2012 11:56 pm
Re: Syzygy DTZ caching in Windows. Waste of memory?
Thanks, and with 64 GB you don't have to worry much about caching I think