MSVC calloc question

[hgm]

I use the call

hashTable = (Bucket*) calloc(nrOfBuckets+1, sizeof(Bucket));

to allocate the hash table in my engine (32-bit-)compiled with gcc -mno-cygwin flag (so linked with msvcrt). Now people complain this returns NULL (resulting in a crash of my engine) when they ask for more than 128MB hash, even though they have 8GB of physical memory on their computer.

Does anyone know how this problem can be solved? They would like to use 1GB hash...

[Joost Buijs]

Normally you can allocate a maximum of 0xFFFFFFFE bytes with calloc() under 32 bit MSVC. I don't know if this also holds for the Express versions of MSVC. I can imagine that they restricted the free compiler in a number of ways.

[hgm]

The MicroSoft docs say that there is a constant _HEAP_MAXREQ, and that calloc will give an error return ENOMEM when you request more than that. It doesn't say, however, how much this is, and my compiler does not seem to know it even when I include windows.h and malloc.h.

It seems to be a problem of the run-time system, though. So if I understand things right, it should not depend on the particular kind of compiler people have, or indeed, if they have any C compiler at all. Even when I have no C compiler, I should be able to run .exe files compiled by someone else, right? It seems Microsoft intentionally sabotage the msvcrt.dll they ship with Windows???

[Joost Buijs]

_HEAP_MAXREQ equals 0xFFFFFFFE so you should be able to allocate 4 gig.
I really don't understand why this ain't working. What version of the MSVC compiler do you use?

[hgm]

I don't use an MSVC compiler. I use gcc under Cygwin. But to my understanding that links with an MSVC library. But as it is a dynamic linking, not with mine, but with the one of the person running the executable.

This is the code I am using:

Code: Select all

			if(mem && memory != mem) {
				memory = mem; mem <<= 20; // MB
				if&#40;hashTable&#41; free&#40;hashTable&#41;;
				for&#40;i= -1; i>mem; i>>=1&#41;; hashMask = &#40;i>>6&#41;;
				hashTable = calloc&#40;hashMask + 2, sizeof&#40;HashBucket&#41;);
				hashTable = &#40;HashBucket*) (&#40;int&#41;hashTable + 63 & ~63&#41;; // align
				if&#40;!hashTable&#41; printf&#40;"tellusererror Could not allocate memory for hash table\n");
				printf&#40;"# hash table %d MB @ %x\n", i+1>>20, hashTable&#41;;
			&#125;

I am not inadvertantly calculating the size wrong, am I? 'i' is an unsigned int, and sizeof(HashBucket) should be 64,and the i>>6 takes care of that.

[Joost Buijs]

Hi Harm Geert,

I tried your code here and with my compiler it works like it should.
The problem has probably something to do with cygwin. I don't have any experience with that.

[hgm]

Well, if I try the same executable on my Windows Vista, it also seems to work like it should. At least, I had no problem allocating 512 MB (and I could see from the task manager that it was indeed used).

[bob]

I use the call

hashTable = (Bucket*) calloc(nrOfBuckets+1, sizeof(Bucket));

to allocate the hash table in my engine (32-bit-)compiled with gcc -mno-cygwin flag (so linked with msvcrt). Now people complain this returns NULL (resulting in a crash of my engine) when they ask for more than 128MB hash, even though they have 8GB of physical memory on their computer.

Does anyone know how this problem can be solved? They would like to use 1GB hash...

Question is, what is the 128m value? is it buckets * sizeof(buckets) or is it just buckets? If so, then 128M * size of buckets might blow a 32 bit value. Or perhaps calloc doesn't like a negative number and that number blows a 31 bit value over into a negative number?

On a 32 bit windows OS can a single process go beyond 4gb anyway? The hardware can address a 36 bit address space thanks to a hack Intel introduced way back. But I don't know how far windows took this to let you go beyond 4 gigs for any single process.

[Gian-Carlo Pascutto]

Old versions of MSVC/Windows were unable to allocate more than 256M with the normal C library calls.

If "more than 128M" means 256M, then alignment or something similar could push it just over 256M.

So the exact version of Windows, and hence, the shipped MSVCRT.DLL is important.

[hgm]

OK, thanks!