Syzygy DTZ caching in Windows. Waste of memory?

Discussion of anything and everything relating to chess playing software and machines.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
Post Reply
User avatar
RJN
Posts: 303
Joined: Fri Jun 21, 2013 3:18 am
Location: Orion Spiral Arm

Syzygy DTZ caching in Windows. Waste of memory?

Post by RJN » Mon May 04, 2015 7:29 pm

I assume Windows caches all files that get read from disk (SSD or HDD) without prejudice. Is there a way to prevent certain file types from being cached? I would think doing so when a DTZ is read is a waste of memory cache capacity. The largest DTZ files are over 2GB.

During a tournament, this wasted cached would then carry over when the next game's WDL are read, with some of the last previous game's WDL kicked out of RAM by the previous game DTZ read once 6 (or 5) pieces had been reached.

If they are cached, what possible way is there to keep *.rtbz from being stored in RAM when they get read off of storage?

Then again, maybe it's better that the DTZ was cached in the previous game while that was being played, so it does not have to be read over and over, despite the fact it should only take a few seconds if DTZ is in a SSD.

Or does Syzygy Bases already take this point into account in some manner?

TIA

syzygy
Posts: 4722
Joined: Tue Feb 28, 2012 10:56 pm

Re: Syzygy DTZ caching in Windows. Waste of memory?

Post by syzygy » Mon May 04, 2015 8:42 pm

RJN wrote:I assume Windows caches all files that get read from disk (SSD or HDD) without prejudice. Is there a way to prevent certain file types from being cached? I would think doing so when a DTZ is read is a waste of memory cache capacity. The largest DTZ files are over 2GB.
They are only accessed at the end of the game, when the 6-piece position is already on the board and the game is effectively over. And then only small parts are accessed and cached. How small those parts is decided by your OS, but an OS is supposed to make reasonable decisions in this respect.

So there is no problem, I would say.

User avatar
RJN
Posts: 303
Joined: Fri Jun 21, 2013 3:18 am
Location: Orion Spiral Arm

Re: Syzygy DTZ caching in Windows. Waste of memory?

Post by RJN » Mon May 04, 2015 8:48 pm

Thanks. I knew they were only accessed at the end of a game. That's why I was wondering about the effect of the next game in a tournament, when the DTZ might have overwritten some WDL cache that is needed in succeeding games.

I guess the fact that only small parts are accessed at a time means it is a non-issue. I had assumed the entire file was read when a DTZ condition happened.

syzygy
Posts: 4722
Joined: Tue Feb 28, 2012 10:56 pm

Re: Syzygy DTZ caching in Windows. Waste of memory?

Post by syzygy » Mon May 04, 2015 8:57 pm

RJN wrote:Thanks. I knew they were only accessed at the end of a game. That's why I was wondering about the effect of the next game in a tournament, when the DTZ might have overwritten some WDL cache that is needed in succeeding games.
If a small part of the WDL cache had to be overwritten, then that just had to happen. The DTZ data did have to be read.
I guess the fact that only small parts are accessed at a time means it is a non-issue. I had assumed the entire file was read when a DTZ condition happened.
Then accessing DTZ files would be really slow (loading 2 GB just to access a few bytes). Memory mapped files are loaded using "demand paging":
http://en.wikipedia.org/wiki/Demand_paging
To understand the third sentence of the initial paragraph you have to know that when a program is launched, its code is not completely loaded from disk into RAM, but memory-mapped into RAM. The relevant code pages are then paged into RAM as the program executes. The same happens with TB files, which are fully memory-mapped into memory on first access, but not fully read from disk.

Memory-mapped data that is loaded into RAM may still be freed again when RAM is needed for other purposes.

User avatar
RJN
Posts: 303
Joined: Fri Jun 21, 2013 3:18 am
Location: Orion Spiral Arm

Re: Syzygy DTZ caching in Windows. Waste of memory?

Post by RJN » Mon May 04, 2015 9:44 pm

syzygy wrote:
RJN wrote:Thanks. I knew they were only accessed at the end of a game. That's why I was wondering about the effect of the next game in a tournament, when the DTZ might have overwritten some WDL cache that is needed in succeeding games.
If a small part of the WDL cache had to be overwritten, then that just had to happen. The DTZ data did have to be read.
I guess the fact that only small parts are accessed at a time means it is a non-issue. I had assumed the entire file was read when a DTZ condition happened.
Then accessing DTZ files would be really slow (loading 2 GB just to access a few bytes). Memory mapped files are loaded using "demand paging":
http://en.wikipedia.org/wiki/Demand_paging
To understand the third sentence of the initial paragraph you have to know that when a program is launched, its code is not completely loaded from disk into RAM, but memory-mapped into RAM. The relevant code pages are then paged into RAM as the program executes. The same happens with TB files, which are fully memory-mapped into memory on first access, but not fully read from disk.

Memory-mapped data that is loaded into RAM may still be freed again when RAM is needed for other purposes.
I'll trust you on that all that, I'm not the best coder :D

Again, thanks for your amazing TBs. An impressive achievement that has become the gold standard of endgame performance. One of the reasons my new machine has 64GB DDR4 are to cache Syzygy bases.

syzygy
Posts: 4722
Joined: Tue Feb 28, 2012 10:56 pm

Re: Syzygy DTZ caching in Windows. Waste of memory?

Post by syzygy » Mon May 04, 2015 9:49 pm

Thanks, and with 64 GB you don't have to worry much about caching I think ;-)

Post Reply