Crafty-23.1 malloc issue

[Dann Corbit]

I have allocated large hash tables in Windows (larger than 2 GB) so I guess it is something POSIX specific.

For Windows, to get proper 64 bit allocation:
EPD Kit revision date: 1996.04.21
unable to open book file [./book.bin].
book is disabled
unable to open book file [./books.bin].

Initializing multiple threads.
System is SMP, not NUMA.

Crafty v23.1 (1 cpus)

White(1): hash 4096M
hash table memory = 4096M bytes.
White(1):

Make this change to utility.c:

Code: Select all

/*
 *******************************************************************************
 *                                                                             *
 *   AlignedMalloc() is used to allocate memory on a precise boundary,         *
 *   primarily to optimize cache performance by forcing the start of the       *
 *   memory region being allocated to match up so that a structure will lie    *
 *   on a single cache line rather than being split across two, assuming the   *
 *   structure is 64 bytes or less of course.                                  *
 *                                                                             *
 *******************************************************************************
 */

void AlignedMalloc(void **pointer, int alignment, size_t size) {
    segments[nsegments][0] = malloc(size + alignment - 1);
    segments[nsegments][1] =
        (void *) (((size_t) segments[nsegments][0] + alignment -
                   1) & ~(alignment - 1));
    *pointer = segments[nsegments][1];
    nsegments++;
}

/*
 *******************************************************************************
 *                                                                             *
 *   AlignedRemalloc() is used to change the size of a memory block that has   *
 *   previously been allocated using AlignedMalloc().                          *
 *                                                                             *
 *******************************************************************************
 */

void AlignedRemalloc(void **pointer, int alignment, size_t size) {
    int i;
    for &#40;i = 0; i < nsegments; i++)
        if &#40;segments&#91;i&#93;&#91;1&#93; == *pointer&#41;
            break;
    if &#40;i == nsegments&#41; &#123;
        Print&#40;4095, "ERROR  AlignedRemalloc&#40;) given an invalid pointer\n");
        exit&#40;1&#41;;
    &#125;
    free&#40;segments&#91;i&#93;&#91;0&#93;);
    segments&#91;i&#93;&#91;0&#93; = malloc&#40;size + alignment - 1&#41;;
    segments&#91;i&#93;&#91;1&#93; =
        &#40;void *) ((&#40;size_t&#41; segments&#91;i&#93;&#91;0&#93; + alignment - 1&#41; & ~&#40;alignment - 1&#41;);
    *pointer = segments&#91;i&#93;&#91;1&#93;;
&#125;

And this change to chess.h

Code: Select all

void AlignedMalloc(void **, int, size_t);
void AlignedRemalloc(void **, int, size_t);

I also make lots of other small changes, but I do not think that they are important for other people. For instance, all paths, such as these:
extern char book_path[128];
extern char log_path[128];
extern char tb_path[128];
extern char rc_path[128];

Get changed to have FILENAME_MAX as their dimension.

and a few other things like that. To do the other changes, I run this macro against all files:
[top][bl]
c\analysis_move[64]\analysis_move[512]\*
[top][bl]
c\annotate_score[100]\annotate_score[512]\*
[top][bl]
c\book_path[128]\book_path[FILENAME_MAX]\*
[top][bl]
c\buffer[100]\buffer[512]\*
[top][bl]
c\buffer[128]\buffer[512]\*
[top][bl]
c\filename[64]\filename[FILENAME_MAX]\*
[top][bl]
c\fname[64]\fname[FILENAME_MAX]\*
[top][bl]
c\history_filename[64]\history_filename[FILENAME_MAX]\*
[top][bl]
c\input[128]\input[512]\*
[top][bl]
c\input_file[100]\input_file[FILENAME_MAX]\*
[top][bl]
c\log_filename[64]\log_filename[FILENAME_MAX]\*
[top][bl]
c\log_path[128]\log_path[FILENAME_MAX]\*
[top][bl]
c\movetext[128]\movetext[512]\*
[top][bl]
c\next[100]\next[512]\*
[top][bl]
c\pname[128]\pname[FILENAME_MAX]\*
[top][bl]
c\prefix[128]\prefix[512]\*
[top][bl]
c\rc_path[128]\rc_path[FILENAME_MAX]\*
[top][bl]
c\tb_path[128]\tb_path[FILENAME_MAX]\*
[top][bl]
c\temp[100]\temp[512]\*
[top][bl]
c\temp[64]\temp[512]\*
[top][bl]
c\text[128]\text[512]\*
[top][bl]
c\title[64]\title[512]\*
[top][bl]
c\value[128]\value[512]\*
[top][bl]
c\data_read[100]\data_read[512]\*
[top][bl]

[bob]

This is a bug. I just re-copied the 23.1 source to my ftp box with a fix. In 23.1, I cleaned up the way things get allocated, and originally used a couple of aligned_malloc() type calls, one for linux, one for windows. Peter reported that the one for windows did not work correctly, even though it was written according to the windows docs for that call. I went back to the old malloc() and force the alignment on a cache block myself, but a couple of size_t types got lost when I undid the windows changes. int won't work as it is not big enough on a 64 bit box, hence the broken malloc. I fixed things and tested with a 16gb hash on a cluster node and it works fine now...

[bob]

For some reason I can't use hash=2048M with crafty-23.1, although I can with crafty-23.0. What am I missing here? I have an 8-core Mac Pro with 6 GB of memory.

Code: Select all

LZsMacPro: ~/Documents/Chess/Crafty/Crafty-23.1] ./crafty-23.1 mt=8 hash=2048M
max threads set to 8.
crafty-23.1(28112) malloc: *** mmap(size=-2147479552) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
AlignedRemalloc() failed, not enough memory.
hash table memory =     0 bytes.

Crafty v23.1 (8 cpus)

White(1): quit


LZsMacPro: ~/Documents/Chess/Crafty/Crafty-23.1] cd ../Crafty-23.0
LZsMacPro: ~/Documents/Chess/Crafty/Crafty-23.0] ./crafty-23.0 mt=8 hash=2048M
max threads set to 8.
hash table memory = 1536M bytes.
max threads set to 8.
EGTB cache memory =  256M bytes.
hash table memory = 3072M bytes.
pawn hash table memory =  512M bytes.
choose from book moves randomly (using weights.)
choose from 5 best moves.
pondering enabled.
Audio output enabled
 game/10 minutes primary time control


Crafty v23.0 (8 cpus)

White(1): quit
LZsMacPro: ~/Documents/Chess/Crafty/Crafty-23.0] 

It is likely a windows issue, related to the way I allocate aligned memory. If you can compile your own, I can send you an attempt at a fix that might work, but I am not sure. I can malloc 12gb on my cluster with no problems.

It works on Windows also. I think it is Mac specific.

See above, it was my error. I just updated the source. I had fixed it once already, but lost the fix when I gave up on the windows aligned malloc() that we could not get to work properly. When I undid that, I lost a couple of critical size_t casts...