SD Syzygy

Discussion of anything and everything relating to chess playing software and machines.

Moderator: Ras

syzygy
Posts: 5774
Joined: Tue Feb 28, 2012 11:56 pm

Re: SD Syzygy

Post by syzygy »

Dirt wrote:
AdminX wrote:Yeah, and I meant 3,4,5 and 6 piece tablebases.
That makes more sense.

Since the files are compressed the CPU has to do some work, which slows things down. I would expect the slower one would be almost as good as the faster one, as long as there are no problems like slowing down from getting too hot.
They are not decompressed to RAM, but cached in the compressed format and decompressed on the fly during the probe. Any reduction of the time that it takes to read a particular disk block into RAM will help.

But whether the faster one is measurably better will depend more on random read performance than on sequential read performance. The numbers given refer to the latter (600x meaning 90 MB/s, 1000x meaning 150 MB/s).
User avatar
MikeB
Posts: 4889
Joined: Thu Mar 09, 2006 6:34 am
Location: Pen Argyl, Pennsylvania

Re: SD Syzygy

Post by MikeB »

I already do this and it works fine - there is no basis - I have seen Crafty hit 40M/nps while probing tbs. Like wise with Stockfish. Of course my machine is different than yours so the ymmv rules come into effect.

According the Geekbench my iMac is ranked 13th out of thousands of benchmarks submitted,
syzygy
Posts: 5774
Joined: Tue Feb 28, 2012 11:56 pm

Re: SD Syzygy

Post by syzygy »

MikeB wrote:I already do this and it works fine - there is no basis - I have seen Crafty hit 40M/nps while probing tbs.
Sure, but you lose 1GB of RAM (or 400 MB if you only store the WDL files). Maybe you would not use that RAM anyway, but in general it's not advisable. And for 6-piece TBs it will really hurt performance.
User avatar
MikeB
Posts: 4889
Joined: Thu Mar 09, 2006 6:34 am
Location: Pen Argyl, Pennsylvania

Re: SD Syzygy

Post by MikeB »

syzygy wrote:
MikeB wrote:I already do this and it works fine - there is no basis - I have seen Crafty hit 40M/nps while probing tbs.
Sure, but you lose 1GB of RAM (or 400 MB if you only store the WDL files). Maybe you would not use that RAM anyway, but in general it's not advisable. And for 6-piece TBs it will really hurt performance.
When you have 32GB of RAM - it's not really a scarce resource on machine.
syzygy
Posts: 5774
Joined: Tue Feb 28, 2012 11:56 pm

Re: SD Syzygy

Post by syzygy »

File Caching
Image
If a program reads data from a file, the OS will first copy (part of) that file into the system (file) cache (page cache on Linux). If the same or another program accesses the same data later, it will read it from the system cache and there is no need to read it from disk. (The left-hand side of the figure does not completely correspond to what happens when accessing TB data. The data is not copied from the system cache into the address space of the process, but the relevant memory pages are directly mapped into the address space.)

Copying files that are often used to a RAM disk will have the effect that the data will be in both the system cache and the RAM disk. It will be present in RAM twice.

Even if the system has lots of RAM, it will now have less RAM left for the system cache. That means that less of the other data and programs that are accessed often can be cached in the system cache.

400 MB on 32 GB total might not be noticeable, but it is clear that the impact will be significant when a RAM disk is used to cache a couple of GB of 6-piece TBs. That means a couple of GBs less for the system cache, which is already under heavy pressure when accessing big portions of the 68 GB of 6-piece WDL data.

Linux has a RAM disk implementation that somehow makes the content of the RAM disk be part of the system cache. That solves the problem of storing the same data twice, but still assumes the user knows better which files are good to cache than the system. I don't know if there is a RAM disk implementation for Windows that is as "smart" as that Linux RAM disk, but I doubt it. Probably only Microsoft could implement one.

In any event, on Linux one can simply do "cat *.rtbw >/dev/null" once and all WDL files are in the system cache (page cache) until the system determines that there is more important data to cache. I don't know if there is an easy way to achieve the same on Windows, but it would be simple to write a small program that sequentially reads all TB data once. Running that program once before launching an engine for the first time after boot would eliminate any initial slowdown that might occur when the 3-4-5-piece TB files are being cached into RAM during probing.
User avatar
MikeB
Posts: 4889
Joined: Thu Mar 09, 2006 6:34 am
Location: Pen Argyl, Pennsylvania

Re: SD Syzygy

Post by MikeB »

syzygy wrote: 400 MB on 32 GB total might not be noticeable, but it is clear that the impact will be significant when a RAM disk is used to cache a couple of GB of 6-piece TBs. That means a couple of GBs less for the system cache, which is already under heavy pressure when accessing big portions of the 68 GB of 6-piece WDL data.

I don't know if there is a RAM disk implementation for Windows that is as "smart" as that Linux RAM disk, but I doubt it. Probably only Microsoft could implement one.
Your point is well taken, but I'm not on Windows and i'm not using 6 man tbs. From an observation standpoint I think it's working pretty well , given my setup. Obviously, ymmv based on your setup..
syzygy
Posts: 5774
Joined: Tue Feb 28, 2012 11:56 pm

Re: SD Syzygy

Post by syzygy »

MikeB wrote:Your point is well taken, but I'm not on Windows and i'm not using 6 man tbs. From an observation standpoint I think it's working pretty well , given my setup. Obviously, ymmv based on your setup..
Ah right. Then I cannot resist remarking that on OS X the

Code: Select all

$ cat /path/to/345TBs/*.rtbw >/dev/null
trick should work as well as on Linux. :-)
BBauer
Posts: 658
Joined: Wed Mar 08, 2006 8:58 pm

Re: SD Syzygy

Post by BBauer »

In windows you may do

Code: Select all

 type *.rtbw >nul
regards
Bernhard
dark_wizzie
Posts: 79
Joined: Mon Jun 30, 2014 3:48 pm
Full name: Erica Lin

Re: SD Syzygy

Post by dark_wizzie »

syzygy wrote:
Dirt wrote:
AdminX wrote:Yeah, and I meant 3,4,5 and 6 piece tablebases.
That makes more sense.

Since the files are compressed the CPU has to do some work, which slows things down. I would expect the slower one would be almost as good as the faster one, as long as there are no problems like slowing down from getting too hot.
They are not decompressed to RAM, but cached in the compressed format and decompressed on the fly during the probe. Any reduction of the time that it takes to read a particular disk block into RAM will help.

But whether the faster one is measurably better will depend more on random read performance than on sequential read performance. The numbers given refer to the latter (600x meaning 90 MB/s, 1000x meaning 150 MB/s).
Hey Ronald, I am curious about the load running Syzygy has on a hard drive. What file size and queue depth are we looking at when the SSD is loading the tablebases?

Thanks
Dark_wizzie, aka Eric Lin. ABK book tester, crappy OTB player. Ex Chess2u mod. Likes pizza.
syzygy
Posts: 5774
Joined: Tue Feb 28, 2012 11:56 pm

Re: SD Syzygy

Post by syzygy »

BBauer wrote:In windows you may do

Code: Select all

 type *.rtbw >nul
Never knew that, thanks!