MSVC calloc question

Discussion of chess software programming and technical issues.

Moderators: hgm, Harvey Williamson, bob

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
User avatar
hgm
Posts: 23086
Joined: Fri Mar 10, 2006 9:06 am
Location: Amsterdam
Full name: H G Muller
Contact:

MSVC calloc question

Post by hgm » Thu Mar 17, 2011 10:51 am

I use the call

hashTable = (Bucket*) calloc(nrOfBuckets+1, sizeof(Bucket));

to allocate the hash table in my engine (32-bit-)compiled with gcc -mno-cygwin flag (so linked with msvcrt). Now people complain this returns NULL (resulting in a crash of my engine) when they ask for more than 128MB hash, even though they have 8GB of physical memory on their computer.

Does anyone know how this problem can be solved? They would like to use 1GB hash...

Joost Buijs
Posts: 813
Joined: Thu Jul 16, 2009 8:47 am
Location: Almere, The Netherlands

Re: MSVC calloc question

Post by Joost Buijs » Thu Mar 17, 2011 12:12 pm

Normally you can allocate a maximum of 0xFFFFFFFE bytes with calloc() under 32 bit MSVC. I don't know if this also holds for the Express versions of MSVC. I can imagine that they restricted the free compiler in a number of ways.

User avatar
hgm
Posts: 23086
Joined: Fri Mar 10, 2006 9:06 am
Location: Amsterdam
Full name: H G Muller
Contact:

Re: MSVC calloc question

Post by hgm » Thu Mar 17, 2011 12:27 pm

The MicroSoft docs say that there is a constant _HEAP_MAXREQ, and that calloc will give an error return ENOMEM when you request more than that. It doesn't say, however, how much this is, and my compiler does not seem to know it even when I include windows.h and malloc.h.

It seems to be a problem of the run-time system, though. So if I understand things right, it should not depend on the particular kind of compiler people have, or indeed, if they have any C compiler at all. Even when I have no C compiler, I should be able to run .exe files compiled by someone else, right? It seems Microsoft intentionally sabotage the msvcrt.dll they ship with Windows???
Last edited by hgm on Thu Mar 17, 2011 12:32 pm, edited 1 time in total.

Joost Buijs
Posts: 813
Joined: Thu Jul 16, 2009 8:47 am
Location: Almere, The Netherlands

Re: MSVC calloc question

Post by Joost Buijs » Thu Mar 17, 2011 12:31 pm

_HEAP_MAXREQ equals 0xFFFFFFFE so you should be able to allocate 4 gig.
I really don't understand why this ain't working. What version of the MSVC compiler do you use?

User avatar
hgm
Posts: 23086
Joined: Fri Mar 10, 2006 9:06 am
Location: Amsterdam
Full name: H G Muller
Contact:

Re: MSVC calloc question

Post by hgm » Thu Mar 17, 2011 12:32 pm

I don't use an MSVC compiler. I use gcc under Cygwin. But to my understanding that links with an MSVC library. But as it is a dynamic linking, not with mine, but with the one of the person running the executable.

This is the code I am using:

Code: Select all

			if(mem && memory != mem) {
				memory = mem; mem <<= 20; // MB
				if&#40;hashTable&#41; free&#40;hashTable&#41;;
				for&#40;i= -1; i>mem; i>>=1&#41;; hashMask = &#40;i>>6&#41;;
				hashTable = calloc&#40;hashMask + 2, sizeof&#40;HashBucket&#41;);
				hashTable = &#40;HashBucket*) (&#40;int&#41;hashTable + 63 & ~63&#41;; // align
				if&#40;!hashTable&#41; printf&#40;"tellusererror Could not allocate memory for hash table\n");
				printf&#40;"# hash table %d MB @ %x\n", i+1>>20, hashTable&#41;;
			&#125;

I am not inadvertantly calculating the size wrong, am I? 'i' is an unsigned int, and sizeof(HashBucket) should be 64,and the i>>6 takes care of that.

Joost Buijs
Posts: 813
Joined: Thu Jul 16, 2009 8:47 am
Location: Almere, The Netherlands

Re: MSVC calloc question

Post by Joost Buijs » Thu Mar 17, 2011 1:29 pm

Hi Harm Geert,

I tried your code here and with my compiler it works like it should.
The problem has probably something to do with cygwin. I don't have any experience with that.

User avatar
hgm
Posts: 23086
Joined: Fri Mar 10, 2006 9:06 am
Location: Amsterdam
Full name: H G Muller
Contact:

Re: MSVC calloc question

Post by hgm » Thu Mar 17, 2011 3:02 pm

Well, if I try the same executable on my Windows Vista, it also seems to work like it should. At least, I had no problem allocating 512 MB (and I could see from the task manager that it was indeed used).

bob
Posts: 20358
Joined: Mon Feb 27, 2006 6:30 pm
Location: Birmingham, AL

Re: MSVC calloc question

Post by bob » Thu Mar 17, 2011 3:40 pm

hgm wrote:I use the call

hashTable = (Bucket*) calloc(nrOfBuckets+1, sizeof(Bucket));

to allocate the hash table in my engine (32-bit-)compiled with gcc -mno-cygwin flag (so linked with msvcrt). Now people complain this returns NULL (resulting in a crash of my engine) when they ask for more than 128MB hash, even though they have 8GB of physical memory on their computer.

Does anyone know how this problem can be solved? They would like to use 1GB hash...
Question is, what is the 128m value? is it buckets * sizeof(buckets) or is it just buckets? If so, then 128M * size of buckets might blow a 32 bit value. Or perhaps calloc doesn't like a negative number and that number blows a 31 bit value over into a negative number?

On a 32 bit windows OS can a single process go beyond 4gb anyway? The hardware can address a 36 bit address space thanks to a hack Intel introduced way back. But I don't know how far windows took this to let you go beyond 4 gigs for any single process.

Gian-Carlo Pascutto
Posts: 1120
Joined: Sat Dec 13, 2008 6:00 pm
Contact:

Re: MSVC calloc question

Post by Gian-Carlo Pascutto » Thu Mar 17, 2011 7:00 pm

Old versions of MSVC/Windows were unable to allocate more than 256M with the normal C library calls.

If "more than 128M" means 256M, then alignment or something similar could push it just over 256M.

So the exact version of Windows, and hence, the shipped MSVCRT.DLL is important.

User avatar
hgm
Posts: 23086
Joined: Fri Mar 10, 2006 9:06 am
Location: Amsterdam
Full name: H G Muller
Contact:

Re: MSVC calloc question

Post by hgm » Thu Mar 17, 2011 7:58 pm

OK, thanks!

Post Reply