Page 1 of 2
MSVC calloc question
Posted: Thu Mar 17, 2011 11:51 am
by hgm
I use the call
hashTable = (Bucket*) calloc(nrOfBuckets+1, sizeof(Bucket));
to allocate the hash table in my engine (32-bit-)compiled with gcc -mno-cygwin flag (so linked with msvcrt). Now people complain this returns NULL (resulting in a crash of my engine) when they ask for more than 128MB hash, even though they have 8GB of physical memory on their computer.
Does anyone know how this problem can be solved? They would like to use 1GB hash...
Re: MSVC calloc question
Posted: Thu Mar 17, 2011 1:12 pm
by Joost Buijs
Normally you can allocate a maximum of 0xFFFFFFFE bytes with calloc() under 32 bit MSVC. I don't know if this also holds for the Express versions of MSVC. I can imagine that they restricted the free compiler in a number of ways.
Re: MSVC calloc question
Posted: Thu Mar 17, 2011 1:27 pm
by hgm
The MicroSoft docs say that there is a constant _HEAP_MAXREQ, and that calloc will give an error return ENOMEM when you request more than that. It doesn't say, however, how much this is, and my compiler does not seem to know it even when I include windows.h and malloc.h.
It seems to be a problem of the run-time system, though. So if I understand things right, it should not depend on the particular kind of compiler people have, or indeed, if they have any C compiler at all. Even when I have no C compiler, I should be able to run .exe files compiled by someone else, right? It seems Microsoft intentionally sabotage the msvcrt.dll they ship with Windows???
Re: MSVC calloc question
Posted: Thu Mar 17, 2011 1:31 pm
by Joost Buijs
_HEAP_MAXREQ equals 0xFFFFFFFE so you should be able to allocate 4 gig.
I really don't understand why this ain't working. What version of the MSVC compiler do you use?
Re: MSVC calloc question
Posted: Thu Mar 17, 2011 1:32 pm
by hgm
I don't use an MSVC compiler. I use gcc under Cygwin. But to my understanding that links with an MSVC library. But as it is a dynamic linking, not with mine, but with the one of the person running the executable.
This is the code I am using:
Code: Select all
if(mem && memory != mem) {
memory = mem; mem <<= 20; // MB
if(hashTable) free(hashTable);
for(i= -1; i>mem; i>>=1); hashMask = (i>>6);
hashTable = calloc(hashMask + 2, sizeof(HashBucket));
hashTable = (HashBucket*) ((int)hashTable + 63 & ~63); // align
if(!hashTable) printf("tellusererror Could not allocate memory for hash table\n");
printf("# hash table %d MB @ %x\n", i+1>>20, hashTable);
}
I am not inadvertantly calculating the size wrong, am I? 'i' is an unsigned int, and sizeof(HashBucket) should be 64,and the i>>6 takes care of that.
Re: MSVC calloc question
Posted: Thu Mar 17, 2011 2:29 pm
by Joost Buijs
Hi Harm Geert,
I tried your code here and with my compiler it works like it should.
The problem has probably something to do with cygwin. I don't have any experience with that.
Re: MSVC calloc question
Posted: Thu Mar 17, 2011 4:02 pm
by hgm
Well, if I try the same executable on my Windows Vista, it also seems to work like it should. At least, I had no problem allocating 512 MB (and I could see from the task manager that it was indeed used).
Re: MSVC calloc question
Posted: Thu Mar 17, 2011 4:40 pm
by bob
hgm wrote:I use the call
hashTable = (Bucket*) calloc(nrOfBuckets+1, sizeof(Bucket));
to allocate the hash table in my engine (32-bit-)compiled with gcc -mno-cygwin flag (so linked with msvcrt). Now people complain this returns NULL (resulting in a crash of my engine) when they ask for more than 128MB hash, even though they have 8GB of physical memory on their computer.
Does anyone know how this problem can be solved? They would like to use 1GB hash...
Question is, what is the 128m value? is it buckets * sizeof(buckets) or is it just buckets? If so, then 128M * size of buckets might blow a 32 bit value. Or perhaps calloc doesn't like a negative number and that number blows a 31 bit value over into a negative number?
On a 32 bit windows OS can a single process go beyond 4gb anyway? The hardware can address a 36 bit address space thanks to a hack Intel introduced way back. But I don't know how far windows took this to let you go beyond 4 gigs for any single process.
Re: MSVC calloc question
Posted: Thu Mar 17, 2011 8:00 pm
by Gian-Carlo Pascutto
Old versions of MSVC/Windows were unable to allocate more than 256M with the normal C library calls.
If "more than 128M" means 256M, then alignment or something similar could push it just over 256M.
So the exact version of Windows, and hence, the shipped MSVCRT.DLL is important.
Re: MSVC calloc question
Posted: Thu Mar 17, 2011 8:58 pm
by hgm
OK, thanks!