Crafty-23.1 malloc issue

[zullil]

For some reason I can't use hash=2048M with crafty-23.1, although I can with crafty-23.0. What am I missing here? I have an 8-core Mac Pro with 6 GB of memory.

Code: Select all

LZsMacPro: ~/Documents/Chess/Crafty/Crafty-23.1] ./crafty-23.1 mt=8 hash=2048M
max threads set to 8.
crafty-23.1(28112) malloc: *** mmap(size=-2147479552) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
AlignedRemalloc() failed, not enough memory.
hash table memory =     0 bytes.

Crafty v23.1 (8 cpus)

White(1): quit


LZsMacPro: ~/Documents/Chess/Crafty/Crafty-23.1] cd ../Crafty-23.0
LZsMacPro: ~/Documents/Chess/Crafty/Crafty-23.0] ./crafty-23.0 mt=8 hash=2048M
max threads set to 8.
hash table memory = 1536M bytes.
max threads set to 8.
EGTB cache memory =  256M bytes.
hash table memory = 3072M bytes.
pawn hash table memory =  512M bytes.
choose from book moves randomly (using weights.)
choose from 5 best moves.
pondering enabled.
Audio output enabled
 game/10 minutes primary time control


Crafty v23.0 (8 cpus)

White(1): quit
LZsMacPro: ~/Documents/Chess/Crafty/Crafty-23.0] 

[Dann Corbit]

I have allocated large hash tables in Windows (larger than 2 GB) so I guess it is something POSIX specific.

[bob]

For some reason I can't use hash=2048M with crafty-23.1, although I can with crafty-23.0. What am I missing here? I have an 8-core Mac Pro with 6 GB of memory.

Code: Select all

LZsMacPro: ~/Documents/Chess/Crafty/Crafty-23.1] ./crafty-23.1 mt=8 hash=2048M
max threads set to 8.
crafty-23.1(28112) malloc: *** mmap(size=-2147479552) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
AlignedRemalloc() failed, not enough memory.
hash table memory =     0 bytes.

Crafty v23.1 (8 cpus)

White(1): quit


LZsMacPro: ~/Documents/Chess/Crafty/Crafty-23.1] cd ../Crafty-23.0
LZsMacPro: ~/Documents/Chess/Crafty/Crafty-23.0] ./crafty-23.0 mt=8 hash=2048M
max threads set to 8.
hash table memory = 1536M bytes.
max threads set to 8.
EGTB cache memory =  256M bytes.
hash table memory = 3072M bytes.
pawn hash table memory =  512M bytes.
choose from book moves randomly (using weights.)
choose from 5 best moves.
pondering enabled.
Audio output enabled
 game/10 minutes primary time control


Crafty v23.0 (8 cpus)

White(1): quit
LZsMacPro: ~/Documents/Chess/Crafty/Crafty-23.0] 

It is likely a windows issue, related to the way I allocate aligned memory. If you can compile your own, I can send you an attempt at a fix that might work, but I am not sure. I can malloc 12gb on my cluster with no problems.

[Dann Corbit]

For some reason I can't use hash=2048M with crafty-23.1, although I can with crafty-23.0. What am I missing here? I have an 8-core Mac Pro with 6 GB of memory.

Code: Select all

LZsMacPro: ~/Documents/Chess/Crafty/Crafty-23.1] ./crafty-23.1 mt=8 hash=2048M
max threads set to 8.
crafty-23.1(28112) malloc: *** mmap(size=-2147479552) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
AlignedRemalloc() failed, not enough memory.
hash table memory =     0 bytes.

Crafty v23.1 (8 cpus)

White(1): quit


LZsMacPro: ~/Documents/Chess/Crafty/Crafty-23.1] cd ../Crafty-23.0
LZsMacPro: ~/Documents/Chess/Crafty/Crafty-23.0] ./crafty-23.0 mt=8 hash=2048M
max threads set to 8.
hash table memory = 1536M bytes.
max threads set to 8.
EGTB cache memory =  256M bytes.
hash table memory = 3072M bytes.
pawn hash table memory =  512M bytes.
choose from book moves randomly (using weights.)
choose from 5 best moves.
pondering enabled.
Audio output enabled
 game/10 minutes primary time control


Crafty v23.0 (8 cpus)

White(1): quit
LZsMacPro: ~/Documents/Chess/Crafty/Crafty-23.0] 

It is likely a windows issue, related to the way I allocate aligned memory. If you can compile your own, I can send you an attempt at a fix that might work, but I am not sure. I can malloc 12gb on my cluster with no problems.

It works on Windows also. I think it is Mac specific.

[Peter Skinner]

It works on Windows also. I think it is Mac specific.

Which version of Windows are you using Dan?

I am using Windows 7 Ultimate 64bit, and I can't use greater than 1024. When trying 2048, I get the following error:

found computer opening book file [./bookc.bin].

Initializing multiple threads.
System is SMP, not NUMA.
max threads set to 4.
EGTB access enabled
using tbpath=./TB
5 piece tablebase files found
14043kb of RAM used for TB indices and decompression tables
AlignedRemalloc() failed, not enough memory.
hash table memory = 0 bytes.
pawn hash table memory = 128M bytes.
EGTB cache memory = 32M bytes.
resign after 5 consecutive moves with score < -6.


Crafty v23.1 (4 cpus)

White(1):

Yet under 23.0:

found computer opening book file [./bookc.bin].

Initializing multiple threads.
System is SMP, not NUMA.
max threads set to 4.
EGTB access enabled
using tbpath=./TB
5 piece tablebase files found
14043kb of RAM used for TB indices and decompression tables
hash table memory = 3072M bytes.
pawn hash table memory = 128M bytes.
EGTB cache memory = 32M bytes.
resign after 5 consecutive moves with score < -6.


Crafty v23.0 (4 cpus)

White(1):

This is definitely a 23.1 issue. Or just my compiles. I will have to look.

Peter

[Aaron Becker]

It works on my mac. I compiled with just "make darwin". Here's output:

crafty-23.1 $ ./crafty hash=2048M
unable to open book file [./book.bin].
book is disabled
unable to open book file [./books.bin].
hash table memory = 2048M bytes.

Crafty v23.1 (1 cpus)

White(1):

[zullil]

It works on my mac. I compiled with just "make darwin". Here's output:

crafty-23.1 $ ./crafty hash=2048M
unable to open book file [./book.bin].
book is disabled
unable to open book file [./books.bin].
hash table memory = 2048M bytes.

Crafty v23.1 (1 cpus)

White(1):

But does it work if you try to make a 64-bit build? Try editing the Makefile to be

Code: Select all

darwin:
        $(MAKE) target=FreeBSD \
                CC=gcc CXX=g++ \
                CFLAGS='$(CFLAGS) -Wall -pipe -O3 -m64' \
                CXFLAGS='-Wall -pipe -O3 -m64' \
                LDFLAGS='$(LDFLAGS) -m64' \
                LIBS='-lpthread -lstdc++' \
                opt='$(opt)' \
                crafty-make

When I build from this I get

Code: Select all

LZsMacPro: ~/Documents/Chess/Crafty/Crafty-23.1] ./crafty hash=2048M
unable to open book file [./book.bin].
book is disabled
unable to open book file [./books.bin].
crafty(30737) malloc: *** mmap(size=-2147479552) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
AlignedRemalloc() failed, not enough memory.
hash table memory =     0 bytes.

Crafty v23.1 (1 cpus)

[Dann Corbit]

Oops, yes. I munge the code.

All the ints in the allocation routines need to be changed into size_t types.

[Aaron Becker]

But does it work if you try to make a 64-bit build?

Good catch. With a 64-bit build I get the same error. The problem is that the 'size' parameter of AlignedMalloc and AlignedRemalloc has type 'int'; it should be a 'size_t'. With that change it works correctly for me. Here's a patch of the changes I made:

diff -rup crafty-23.1/chess.h crafty-23.1-fixed/chess.h
--- crafty-23.1/chess.h 2009-11-18 12:43:12.000000000 -0600
+++ crafty-23.1-fixed/chess.h 2009-11-20 20:15:20.000000000 -0600
@@ -23,7 +23,6 @@
#include <stdio.h>
#include <assert.h>
#include <stdlib.h>
-#include <malloc.h>
#include <string.h>
#if !defined(TYPES_INCLUDED)
# include "lock.h"
@@ -375,8 +374,8 @@ int CDECL MSB(BITBOARD);
int CDECL LSB(BITBOARD);
# endif
# endif
-void AlignedMalloc(void **, int, int);
-void AlignedRemalloc(void **, int, int);
+void AlignedMalloc(void **, int, size_t);
+void AlignedRemalloc(void **, int, size_t);
void Analyze(void);
void Annotate(void);
void AnnotateHeaderHTML(char *, FILE *);
Only in crafty-23.1-fixed/: game.001
Only in crafty-23.1-fixed/: log.001
diff -rup crafty-23.1/utility.c crafty-23.1-fixed/utility.c
--- crafty-23.1/utility.c 2009-11-18 12:43:12.000000000 -0600
+++ crafty-23.1-fixed/utility.c 2009-11-20 20:14:56.000000000 -0600
@@ -67,7 +67,7 @@
*******************************************************************************
*/

-void AlignedMalloc(void **pointer, int alignment, int size) {
+void AlignedMalloc(void **pointer, int alignment, size_t size) {
segments[nsegments][0] = malloc(size + alignment - 1);
segments[nsegments][1] =
(void *) (((long) segments[nsegments][0] + alignment -
@@ -85,7 +85,7 @@ void AlignedMalloc(void **pointer, int a
*******************************************************************************
*/

-void AlignedRemalloc(void **pointer, int alignment, int size) {
+void AlignedRemalloc(void **pointer, int alignment, size_t size) {
int i;
for (i = 0; i < nsegments; i++)
if (segments[1] == *pointer)

[Dann Corbit]

I have allocated large hash tables in Windows (larger than 2 GB) so I guess it is something POSIX specific.

For Windows, to get proper 64 bit allocation:
EPD Kit revision date: 1996.04.21
unable to open book file [./book.bin].
book is disabled
unable to open book file [./books.bin].

Initializing multiple threads.
System is SMP, not NUMA.

Crafty v23.1 (1 cpus)

White(1): hash 4096M
hash table memory = 4096M bytes.
White(1):

Make this change to utility.c:

Code: Select all

/*
 *******************************************************************************
 *                                                                             *
 *   AlignedMalloc() is used to allocate memory on a precise boundary,         *
 *   primarily to optimize cache performance by forcing the start of the       *
 *   memory region being allocated to match up so that a structure will lie    *
 *   on a single cache line rather than being split across two, assuming the   *
 *   structure is 64 bytes or less of course.                                  *
 *                                                                             *
 *******************************************************************************
 */

void AlignedMalloc(void **pointer, int alignment, size_t size) {
    segments[nsegments][0] = malloc(size + alignment - 1);
    segments[nsegments][1] =
        (void *) (((size_t) segments[nsegments][0] + alignment -
                   1) & ~(alignment - 1));
    *pointer = segments[nsegments][1];
    nsegments++;
}

/*
 *******************************************************************************
 *                                                                             *
 *   AlignedRemalloc() is used to change the size of a memory block that has   *
 *   previously been allocated using AlignedMalloc().                          *
 *                                                                             *
 *******************************************************************************
 */

void AlignedRemalloc(void **pointer, int alignment, size_t size) {
    int i;
    for (i = 0; i < nsegments; i++)
        if (segments[i][1] == *pointer)
            break;
    if (i == nsegments) {
        Print(4095, "ERROR  AlignedRemalloc() given an invalid pointer\n");
        exit(1);
    }
    free(segments[i][0]);
    segments[i][0] = malloc(size + alignment - 1);
    segments[i][1] =
        (void *) (((size_t) segments[i][0] + alignment - 1) & ~(alignment - 1));
    *pointer = segments[i][1];
}

And this change to chess.h

Code: Select all

void AlignedMalloc(void **, int, size_t);
void AlignedRemalloc(void **, int, size_t);