malloc more than 2GB ?

Discussion of chess software programming and technical issues.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
Post Reply
frankp
Posts: 224
Joined: Sun Mar 12, 2006 2:11 pm

malloc more than 2GB ?

Post by frankp » Sat Mar 28, 2009 6:09 pm

gcc version 4.3.2 (Debian 4.3.2-1.1)

On a 64bit system is it possible to get gcc to malloc more than 2GB, which seems to be the limit?

Volker Annuss
Posts: 173
Joined: Mon Sep 03, 2007 7:15 am

Re: malloc more than 2GB ?

Post by Volker Annuss » Sat Mar 28, 2009 6:38 pm

Hello Frank,

it works here with Ubuntu 8.10, gcc 4.3.2.

Code: Select all

#include <stdlib.h>
#include <stdio.h>

int main&#40; int argc, char *argv&#91;&#93; )
&#123;
	void *p = malloc&#40; 0x100000001LL );
	printf&#40; "%p\n", p );
	exit&#40; 0 );
&#125;

frankp
Posts: 224
Joined: Sun Mar 12, 2006 2:11 pm

Re: malloc more than 2GB ?

Post by frankp » Sat Mar 28, 2009 8:00 pm

Thanks. It works for me too.

I forgot I was making space for a record each of 12 bytes. How embarrassing.....

frankp
Posts: 224
Joined: Sun Mar 12, 2006 2:11 pm

Re: malloc more than 2GB ?

Post by frankp » Sat Mar 28, 2009 8:10 pm

frankp wrote:Thanks. It works for me too.

I forgot I was making space for a record each of 12 bytes. How embarrassing.....
Perhaps not.

I have effectively:

num = (memory) / sizeof (recordT);
p = (recordT*) malloc (num * sizeof(recordT));

If memory is >2GB then if fails - NULL returned.

User avatar
Zach Wegner
Posts: 1922
Joined: Wed Mar 08, 2006 11:51 pm
Location: Earth
Contact:

Re: malloc more than 2GB ?

Post by Zach Wegner » Sat Mar 28, 2009 8:51 pm

frankp wrote:
frankp wrote:Thanks. It works for me too.

I forgot I was making space for a record each of 12 bytes. How embarrassing.....
Perhaps not.

I have effectively:

num = (memory) / sizeof (recordT);
p = (recordT*) malloc (num * sizeof(recordT));

If memory is >2GB then if fails - NULL returned.
Is memory a signed int perchance? Then you would be trying to malloc negative memory.

frankp
Posts: 224
Joined: Sun Mar 12, 2006 2:11 pm

Re: malloc more than 2GB ?

Post by frankp » Sat Mar 28, 2009 9:17 pm

No...
I tried num = (3000 * 1000000) / sizeof (recordT), where num is uint64

num = (2000 * 1000000) / sizeof (recordT), follow by the malloc is fine.

bob
Posts: 20916
Joined: Mon Feb 27, 2006 6:30 pm
Location: Birmingham, AL

Re: malloc more than 2GB ?

Post by bob » Sat Mar 28, 2009 10:25 pm

frankp wrote:gcc version 4.3.2 (Debian 4.3.2-1.1)

On a 64bit system is it possible to get gcc to malloc more than 2GB, which seems to be the limit?
Doesn't seem to be an issue on my linux-64 boxes:

White(1): hash=8192m
hash table memory = 6144M bytes.

Been running with big hashes in things like CCT for years.

User avatar
Zach Wegner
Posts: 1922
Joined: Wed Mar 08, 2006 11:51 pm
Location: Earth
Contact:

Re: malloc more than 2GB ?

Post by Zach Wegner » Sun Mar 29, 2009 12:01 am

frankp wrote:No...
I tried num = (3000 * 1000000) / sizeof (recordT), where num is uint64
If you had it exactly as written, it would cause issues. The integer literals are signed ints, so the multiplication gets performed (which overflows the signed int and is thus negative), then the division, and then when the whole thing is casted to an unsigned 64 bit by sign extension. The result is still negative.

User avatar
Zach Wegner
Posts: 1922
Joined: Wed Mar 08, 2006 11:51 pm
Location: Earth
Contact:

Re: malloc more than 2GB ?

Post by Zach Wegner » Sun Mar 29, 2009 2:34 am

Oops, of course, the result is not negative. But the result is sign extended to 64 bits, which gives 2^64-3 billion. I just confirmed this with an actual program...

frankp
Posts: 224
Joined: Sun Mar 12, 2006 2:11 pm

Re: malloc more than 2GB ?

Post by frankp » Sun Mar 29, 2009 9:49 am

Zach Wegner wrote:Oops, of course, the result is not negative. But the result is sign extended to 64 bits, which gives 2^64-3 billion. I just confirmed this with an actual program...
Thanks Zach. I just cast the numbers to unsigned and it works fine.

Post Reply