Nalimov and memory for indexes (are you aware?)

Discussion of anything and everything relating to chess playing software and machines.

Moderators: hgm, Harvey Williamson, bob

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
User avatar
michiguel
Posts: 6386
Joined: Thu Mar 09, 2006 7:30 pm
Location: Chicago, Illinois, USA
Contact:

Nalimov and memory for indexes (are you aware?)

Post by michiguel » Mon Mar 01, 2010 9:18 pm

Is the people aware the Nalimov take ~20 MiB (as I was told in another forum, correct me if I am wrong) at start up to load indexes?
How do you take this into account when you do testing? is it part of the memory that you give to the engines + egtb cache? if you give an engine 20 MiB + 32 cache that would be significant.

Gaviota is taking 9.8 MiB and bothers me. I am looking to see how I reduce it (I can cache some and compress the other).

Miguel

BrandonSi

Re: Nalimov and memory for indexes (are you aware?)

Post by BrandonSi » Mon Mar 01, 2010 9:24 pm

Hey neighbor :)

Is this 20MB assumption for 3-4-5 tablebases?
is it part of the memory that you give to the engines + egtb cache?
I can't state this as a certainty for every situation, but for engines running under the Chessbase GUI, tablebase memory is specified (and I assume allocated) separately from the memory allocated to the engine and its hash files / table.

Gian-Carlo Pascutto
Posts: 1120
Joined: Sat Dec 13, 2008 6:00 pm
Contact:

Re: Nalimov and memory for indexes (are you aware?)

Post by Gian-Carlo Pascutto » Mon Mar 01, 2010 9:26 pm

What you specify in the GUI for Nalimov is the cache.

He's talking about the indexes, which you cannot configure. It gets much larger than 20Mb if you use 6-men tables.

BrandonSi

Re: Nalimov and memory for indexes (are you aware?)

Post by BrandonSi » Mon Mar 01, 2010 9:43 pm

Gian-Carlo Pascutto wrote:What you specify in the GUI for Nalimov is the cache.

He's talking about the indexes, which you cannot configure. It gets much larger than 20Mb if you use 6-men tables.
Ah, I understand. However, since ultimately any bottleneck associated with Nalimov bases are going to be limited by the physical disk / data path, and not all indexes would be loaded into memory at the same time, I'm not sure I understand why the space requirement in RAM for the indexes is an issue...?

Just curious. :)

User avatar
michiguel
Posts: 6386
Joined: Thu Mar 09, 2006 7:30 pm
Location: Chicago, Illinois, USA
Contact:

Re: Nalimov and memory for indexes (are you aware?)

Post by michiguel » Mon Mar 01, 2010 11:01 pm

BrandonSi wrote:
Gian-Carlo Pascutto wrote:What you specify in the GUI for Nalimov is the cache.

He's talking about the indexes, which you cannot configure. It gets much larger than 20Mb if you use 6-men tables.
Ah, I understand. However, since ultimately any bottleneck associated with Nalimov bases are going to be limited by the physical disk / data path, and not all indexes would be loaded into memory at the same time, I'm not sure I understand why the space requirement in RAM for the indexes is an issue...?

Just curious. :)
As a matter of principle, it is not a good idea that an engine takes memory without letting you know. For instance, in a test, you give equal RAM to two engines, but one of them is getting 20 MiB extra. That might not seem a lot in current high end hardware, but you always find pocketPC, netbooks, phones etc. in which 20 MiB is a lot.

Second, the problem is scalable when you want to use 6-piece TBs, for instance, as GCP mentioned.

The appropiate way to handle this is to subtract this memory from the cache. If you tell the engine to get 32 MiB of EGTB cache, then 20 of those should be reserved for the indexes. That would keep the engine "honest" regarding its requirements.

I just wanted to know if the people took this into account or not.

Miguel

User avatar
Harvey Williamson
Posts: 1760
Joined: Sun May 25, 2008 9:12 pm
Location: Media City, UK
Contact:

Re: Nalimov and memory for indexes (are you aware?)

Post by Harvey Williamson » Mon Mar 01, 2010 11:11 pm

michiguel wrote:
BrandonSi wrote:
Gian-Carlo Pascutto wrote:What you specify in the GUI for Nalimov is the cache.

He's talking about the indexes, which you cannot configure. It gets much larger than 20Mb if you use 6-men tables.
Ah, I understand. However, since ultimately any bottleneck associated with Nalimov bases are going to be limited by the physical disk / data path, and not all indexes would be loaded into memory at the same time, I'm not sure I understand why the space requirement in RAM for the indexes is an issue...?

Just curious. :)
As a matter of principle, it is not a good idea that an engine takes memory without letting you know. For instance, in a test, you give equal RAM to two engines, but one of them is getting 20 MiB extra. That might not seem a lot in current high end hardware, but you always find pocketPC, netbooks, phones etc. in which 20 MiB is a lot.

Second, the problem is scalable when you want to use 6-piece TBs, for instance, as GCP mentioned.

The appropiate way to handle this is to subtract this memory from the cache. If you tell the engine to get 32 MiB of EGTB cache, then 20 of those should be reserved for the indexes. That would keep the engine "honest" regarding its requirements.

I just wanted to know if the people took this into account or not.

Miguel
Rybka is very bad at this whatever you set as tb cache it will use x the number of cores so on an 8 core machine if you set cache at 64 Rybka will take 8x64 - i have not seen other engines do this.

User avatar
Werner
Posts: 2334
Joined: Wed Mar 08, 2006 9:09 pm

Re: Nalimov and memory for indexes (are you aware?)

Post by Werner » Tue Mar 02, 2010 7:51 am

Hi,
I think in this case Windows task manager does not show it correct. If you look at the rest of free memory you see only 1 times the 64 MB are used.
Werner

Jouni
Posts: 1949
Joined: Wed Mar 08, 2006 7:15 pm

Re: Nalimov and memory for indexes (are you aware?)

Post by Jouni » Tue Mar 02, 2010 10:26 am

20 MB is nothing, when R3 uses 90 MB always for nothing :)

Jouni

Gian-Carlo Pascutto
Posts: 1120
Joined: Sat Dec 13, 2008 6:00 pm
Contact:

Re: Nalimov and memory for indexes (are you aware?)

Post by Gian-Carlo Pascutto » Tue Mar 02, 2010 1:08 pm

BrandonSi wrote:not all indexes would be loaded into memory at the same time,
They are. It's limitation of the Nalimov code.

Gian-Carlo Pascutto
Posts: 1120
Joined: Sat Dec 13, 2008 6:00 pm
Contact:

Re: Nalimov and memory for indexes (are you aware?)

Post by Gian-Carlo Pascutto » Tue Mar 02, 2010 1:09 pm

Harvey Williamson wrote: Rybka is very bad at this whatever you set as tb cache it will use x the number of cores so on an 8 core machine if you set cache at 64 Rybka will take 8x64 - i have not seen other engines do this.
Probably an issue with every engine that is multiprocessed. Zappa will likely be affected too.

It's not only the caches, the indexes will also get replicated. Nalimov wrote his code for Crafty at the moment Crafty was multithreaded, and the code just sucks for multiprocessed engines.
Last edited by Gian-Carlo Pascutto on Tue Mar 02, 2010 1:11 pm, edited 1 time in total.

Post Reply