Nalimov and memory for indexes (are you aware?)

[Gian-Carlo Pascutto]

Hi,
I think in this case Windows task manager does not show it correct. If you look at the rest of free memory you see only 1 times the 64 MB are used.

For the hashtables that is the case, but for Nalimov it's not. Unless Vasik rewrote all the Nalimov code to work with shared memory, which I simply don't believe.

[M ANSARI]

Well I have to agree that there is something different with R3 and Nalimov memory usage. I noticed that when I put EGTB's on a USB drive it takes ages for the engine to load and unload. For some reason if EGTB's are on HDD then this is not a problem. Once loaded things are OK. I don't see this behaviour with other engines and to be honest I have never figured this one out.

[michiguel]

Rybka is very bad at this whatever you set as tb cache it will use x the number of cores so on an 8 core machine if you set cache at 64 Rybka will take 8x64 - i have not seen other engines do this.

Probably an issue with every engine that is multiprocessed. Zappa will likely be affected too.

It's not only the caches, the indexes will also get replicated. Nalimov wrote his code for Crafty at the moment Crafty was multithreaded, and the code just sucks for multiprocessed engines.

I guess that at least they could ask for EGTB Cache/core rather than EGTB Cache, to let the user know what is going on.

Anyway, I am very interested in your expert opinion:

I think it is not a good idea to have (for instance) 4 caches of 32 MiB, one for each thread. It would be better to have a 128 MiB cache shared by all threads, and properly protected for read/write operations (that is what I am doing currently with Gaviota TBs). Of course, this creates a problem when all threads start to hit the cache and it could potentially degrade the parallel scalability. However, the EGTB probe is dominated so much by HD access than anything else is almost irrelevant. Having a bigger shared cache decreases significantly the likelihood of HD access. I prefer to reduce HD accesses rather than the potential overlap of two threads trying to access the EGTB cache. Most of those problems are previously faced by the hash table probe already.

What do you think?

Miguel

[michiguel]

Well I have to agree that there is something different with R3 and Nalimov memory usage. I noticed that when I put EGTB's on a USB drive it takes ages for the engine to load and unload. For some reason if EGTB's are on HDD then this is not a problem. Once loaded things are OK. I don't see this behaviour with other engines and to be honest I have never figured this one out.

Each of the threads load their own indexes? Then it is 20 MiB x cores?

Miguel

[M ANSARI]

Rybka uses processes and not threads and I think that is what causes some problems. I forgot to mention that the problem mentioned earlier with slow loading and unloading of engines when USB drive is used for EGTB is a problem only with my Octa's.

[Gian-Carlo Pascutto]

Well I have to agree that there is something different with R3 and Nalimov memory usage. I noticed that when I put EGTB's on a USB drive it takes ages for the engine to load and unload. For some reason if EGTB's are on HDD then this is not a problem. Once loaded things are OK. I don't see this behaviour with other engines and to be honest I have never figured this one out.

Each of the threads load their own indexes? Then it is 20 MiB x cores?

Miguel

Each *process*, see my comment above.

There's a well-known bug where SMP Rybka doesn't use tablebases correctly, and it's closely related: none of the Nalimov stuff is shared and Vasik forgot to pass a parameter from the master process to the slaves.

I'm glad I got rid of sh*t like that in DS 3.0

[Gian-Carlo Pascutto]

Anyway, I am very interested in your expert opinion:

I think it is not a good idea to have (for instance) 4 caches of 32 MiB, one for each thread. It would be better to have a 128 MiB cache shared by all threads, and properly protected for read/write operations (that is what I am doing currently with Gaviota TBs). Of course, this creates a problem when all threads start to hit the cache and it could potentially degrade the parallel scalability. However, the EGTB probe is dominated so much by HD access than anything else is almost irrelevant. Having a bigger shared cache decreases significantly the likelihood of HD access. I prefer to reduce HD accesses rather than the potential overlap of two threads trying to access the EGTB cache. Most of those problems are previously faced by the hash table probe already.

What do you think?

Miguel

Of course you are right. Nalimov also knew this, which is why he made his code and cache SMP safe exactly as you describe.

The problem for some programmers is that the code was meant for *multithreaded* engines where everything is shared by default, and they are *multiprocessed* where nothing is shared by default.

IIRC, Nalimov allows you to allocate the cache yourself, so you could do that in shared memory, but this doesn't solve the problem with the indexes at all.

[Harvey Williamson]

Hi,
I think in this case Windows task manager does not show it correct. If you look at the rest of free memory you see only 1 times the 64 MB are used.

For tb cache it is correct - I asked Vas about this and he confirmed it.

[Zach Wegner]

Rybka is very bad at this whatever you set as tb cache it will use x the number of cores so on an 8 core machine if you set cache at 64 Rybka will take 8x64 - i have not seen other engines do this.

Probably an issue with every engine that is multiprocessed. Zappa will likely be affected too.

It's not only the caches, the indexes will also get replicated. Nalimov wrote his code for Crafty at the moment Crafty was multithreaded, and the code just sucks for multiprocessed engines.

Zappa uses threads.

[Gian-Carlo Pascutto]

Ok. According to the chessprogramming Wiki, you, Vincent and Vasik are the lone holdouts, then.