Syzygy DTZ caching in Windows. Waste of memory?

[RJN]

I assume Windows caches all files that get read from disk (SSD or HDD) without prejudice. Is there a way to prevent certain file types from being cached? I would think doing so when a DTZ is read is a waste of memory cache capacity. The largest DTZ files are over 2GB.

During a tournament, this wasted cached would then carry over when the next game's WDL are read, with some of the last previous game's WDL kicked out of RAM by the previous game DTZ read once 6 (or 5) pieces had been reached.

If they are cached, what possible way is there to keep *.rtbz from being stored in RAM when they get read off of storage?

Then again, maybe it's better that the DTZ was cached in the previous game while that was being played, so it does not have to be read over and over, despite the fact it should only take a few seconds if DTZ is in a SSD.

Or does Syzygy Bases already take this point into account in some manner?

TIA

[syzygy]

I assume Windows caches all files that get read from disk (SSD or HDD) without prejudice. Is there a way to prevent certain file types from being cached? I would think doing so when a DTZ is read is a waste of memory cache capacity. The largest DTZ files are over 2GB.

They are only accessed at the end of the game, when the 6-piece position is already on the board and the game is effectively over. And then only small parts are accessed and cached. How small those parts is decided by your OS, but an OS is supposed to make reasonable decisions in this respect.

So there is no problem, I would say.

[RJN]

Thanks. I knew they were only accessed at the end of a game. That's why I was wondering about the effect of the next game in a tournament, when the DTZ might have overwritten some WDL cache that is needed in succeeding games.

I guess the fact that only small parts are accessed at a time means it is a non-issue. I had assumed the entire file was read when a DTZ condition happened.

[syzygy]

Thanks. I knew they were only accessed at the end of a game. That's why I was wondering about the effect of the next game in a tournament, when the DTZ might have overwritten some WDL cache that is needed in succeeding games.

If a small part of the WDL cache had to be overwritten, then that just had to happen. The DTZ data did have to be read.

I guess the fact that only small parts are accessed at a time means it is a non-issue. I had assumed the entire file was read when a DTZ condition happened.

Then accessing DTZ files would be really slow (loading 2 GB just to access a few bytes). Memory mapped files are loaded using "demand paging":
http://en.wikipedia.org/wiki/Demand_paging
To understand the third sentence of the initial paragraph you have to know that when a program is launched, its code is not completely loaded from disk into RAM, but memory-mapped into RAM. The relevant code pages are then paged into RAM as the program executes. The same happens with TB files, which are fully memory-mapped into memory on first access, but not fully read from disk.

Memory-mapped data that is loaded into RAM may still be freed again when RAM is needed for other purposes.

[RJN]

Thanks. I knew they were only accessed at the end of a game. That's why I was wondering about the effect of the next game in a tournament, when the DTZ might have overwritten some WDL cache that is needed in succeeding games.

If a small part of the WDL cache had to be overwritten, then that just had to happen. The DTZ data did have to be read.

I guess the fact that only small parts are accessed at a time means it is a non-issue. I had assumed the entire file was read when a DTZ condition happened.

Then accessing DTZ files would be really slow (loading 2 GB just to access a few bytes). Memory mapped files are loaded using "demand paging":
http://en.wikipedia.org/wiki/Demand_paging
To understand the third sentence of the initial paragraph you have to know that when a program is launched, its code is not completely loaded from disk into RAM, but memory-mapped into RAM. The relevant code pages are then paged into RAM as the program executes. The same happens with TB files, which are fully memory-mapped into memory on first access, but not fully read from disk.

Memory-mapped data that is loaded into RAM may still be freed again when RAM is needed for other purposes.

I'll trust you on that all that, I'm not the best coder

Again, thanks for your amazing TBs. An impressive achievement that has become the gold standard of endgame performance. One of the reasons my new machine has 64GB DDR4 are to cache Syzygy bases.

[syzygy]

Thanks, and with 64 GB you don't have to worry much about caching I think