Nalimov and memory for indexes (are you aware?)

[michiguel]

Is the people aware the Nalimov take ~20 MiB (as I was told in another forum, correct me if I am wrong) at start up to load indexes?
How do you take this into account when you do testing? is it part of the memory that you give to the engines + egtb cache? if you give an engine 20 MiB + 32 cache that would be significant.

Gaviota is taking 9.8 MiB and bothers me. I am looking to see how I reduce it (I can cache some and compress the other).

Miguel

[BrandonSi]

Hey neighbor

Is this 20MB assumption for 3-4-5 tablebases?

is it part of the memory that you give to the engines + egtb cache?

I can't state this as a certainty for every situation, but for engines running under the Chessbase GUI, tablebase memory is specified (and I assume allocated) separately from the memory allocated to the engine and its hash files / table.

[Gian-Carlo Pascutto]

What you specify in the GUI for Nalimov is the cache.

He's talking about the indexes, which you cannot configure. It gets much larger than 20Mb if you use 6-men tables.

[BrandonSi]

What you specify in the GUI for Nalimov is the cache.

He's talking about the indexes, which you cannot configure. It gets much larger than 20Mb if you use 6-men tables.

Ah, I understand. However, since ultimately any bottleneck associated with Nalimov bases are going to be limited by the physical disk / data path, and not all indexes would be loaded into memory at the same time, I'm not sure I understand why the space requirement in RAM for the indexes is an issue...?

Just curious.

[michiguel]

What you specify in the GUI for Nalimov is the cache.

He's talking about the indexes, which you cannot configure. It gets much larger than 20Mb if you use 6-men tables.

Ah, I understand. However, since ultimately any bottleneck associated with Nalimov bases are going to be limited by the physical disk / data path, and not all indexes would be loaded into memory at the same time, I'm not sure I understand why the space requirement in RAM for the indexes is an issue...?

Just curious.

As a matter of principle, it is not a good idea that an engine takes memory without letting you know. For instance, in a test, you give equal RAM to two engines, but one of them is getting 20 MiB extra. That might not seem a lot in current high end hardware, but you always find pocketPC, netbooks, phones etc. in which 20 MiB is a lot.

Second, the problem is scalable when you want to use 6-piece TBs, for instance, as GCP mentioned.

The appropiate way to handle this is to subtract this memory from the cache. If you tell the engine to get 32 MiB of EGTB cache, then 20 of those should be reserved for the indexes. That would keep the engine "honest" regarding its requirements.

I just wanted to know if the people took this into account or not.

Miguel

[Harvey Williamson]

What you specify in the GUI for Nalimov is the cache.

He's talking about the indexes, which you cannot configure. It gets much larger than 20Mb if you use 6-men tables.

Ah, I understand. However, since ultimately any bottleneck associated with Nalimov bases are going to be limited by the physical disk / data path, and not all indexes would be loaded into memory at the same time, I'm not sure I understand why the space requirement in RAM for the indexes is an issue...?

Just curious.

As a matter of principle, it is not a good idea that an engine takes memory without letting you know. For instance, in a test, you give equal RAM to two engines, but one of them is getting 20 MiB extra. That might not seem a lot in current high end hardware, but you always find pocketPC, netbooks, phones etc. in which 20 MiB is a lot.

Second, the problem is scalable when you want to use 6-piece TBs, for instance, as GCP mentioned.

The appropiate way to handle this is to subtract this memory from the cache. If you tell the engine to get 32 MiB of EGTB cache, then 20 of those should be reserved for the indexes. That would keep the engine "honest" regarding its requirements.

I just wanted to know if the people took this into account or not.

Miguel

Rybka is very bad at this whatever you set as tb cache it will use x the number of cores so on an 8 core machine if you set cache at 64 Rybka will take 8x64 - i have not seen other engines do this.

[Werner]

Hi,
I think in this case Windows task manager does not show it correct. If you look at the rest of free memory you see only 1 times the 64 MB are used.

[Jouni]

20 MB is nothing, when R3 uses 90 MB always for nothing

Jouni

[Gian-Carlo Pascutto]

not all indexes would be loaded into memory at the same time,

They are. It's limitation of the Nalimov code.

[Gian-Carlo Pascutto]

Rybka is very bad at this whatever you set as tb cache it will use x the number of cores so on an 8 core machine if you set cache at 64 Rybka will take 8x64 - i have not seen other engines do this.

Probably an issue with every engine that is multiprocessed. Zappa will likely be affected too.

It's not only the caches, the indexes will also get replicated. Nalimov wrote his code for Crafty at the moment Crafty was multithreaded, and the code just sucks for multiprocessed engines.