NUMA 101

[bob]

Very useful, thanks.

When I have looked at this before, a couple complications have come up:

1. It seems that Windows and POSIX-type OSs have completely different APIs for this stuff, with different capabilities.

2. Determining the exact CPU architecture you have is apparently quite tricky. So when you pin to a CPU, what exactly does that mean? I always disable hyperthreading on my machines, so I only want to consider physical cores, not virtual ones. But many people run with hyperthreading on. Also, recent AMD architectures have 2 NUMA nodes within a single physical CPU.

Any advice on handling this stuff across varying OSs and architectures?

--Jon

First, I never have hyper threading enabled. It is painful to try to determine which processors share the same physical core, and there is no rule that says cpu 0 and 1 share a core, etc. That's a software issue that is painful. I have seen 0 and 1 share a core, 0-N sharing cores with N+1 to 2N. I disable that nonsense to avoid the hassle.

Cross-platform (I assume you mainly address linux vs windows) has ALWAYS been a PITA. Even creating threads is different, much less basic things like select() and such. Windows has some neat NUMA stuff, as does Linux. And of course they are 100% incompatible.

[zullil]

First, since not all machines support NUMA, I have a -DNUMA Makefile option that turns this on. Leave -DNUMA off, and it doesn't do any NUMA-related tricks at all.

Including -DNUMA leads to the following error. This is on a linux system:

Code: Select all

$ make profile
make -j unix-gcc-profile
make[1]: Entering directory `/home/louis/Documents/Chess/Crafty'
make -j target=UNIX \
		CC=gcc-5 CXX=g++-5 \
		opt='-DTEST -DINLINEASM -DPOPCNT -DCPUS=20 -DAFFINITY -DNUMA' \
		CFLAGS='-Wall -Wno-array-bounds -pipe -O3 -march=native -fprofile-arcs \
		-pthread' \
		CXFLAGS='-Wall -Wno-array-bounds -pipe -O3 -march=native -fprofile-arcs \
		-pthread' \
		LDFLAGS=' -fprofile-arcs -pthread -lstdc++ ' \
		crafty-make
make[2]: Entering directory `/home/louis/Documents/Chess/Crafty'
make[3]: Entering directory `/home/louis/Documents/Chess/Crafty'
gcc-5 -Wall -Wno-array-bounds -pipe -O3 -march=native -fprofile-arcs \
-pthread -DTEST -DINLINEASM -DPOPCNT -DCPUS=20 -DAFFINITY -DNUMA -DUNIX -c crafty.c
g++-5 -c -Wall -Wno-array-bounds -pipe -O3 -march=native -fprofile-arcs \
-pthread -DTEST -DINLINEASM -DPOPCNT -DCPUS=20 -DAFFINITY -DNUMA -DUNIX egtb.cpp
In file included from crafty.c:45:0:
main.c: In function ‘main’:
main.c:4309:26: warning: passing argument 2 of ‘numa_node_to_cpus’ from incompatible pointer type [-Wincompatible-pointer-types]
     numa_node_to_cpus(0, cpus, 64);
                          ^
In file included from main.c:9:0,
                 from crafty.c:45:
/usr/include/numa.h:283:5: note: expected ‘struct bitmask *’ but argument is of type ‘long unsigned int *’
 int numa_node_to_cpus(int, struct bitmask *);
     ^
In file included from crafty.c:45:0:
main.c:4309:5: error: too many arguments to function ‘numa_node_to_cpus’
     numa_node_to_cpus(0, cpus, 64);
     ^
In file included from main.c:9:0,
                 from crafty.c:45:
/usr/include/numa.h:283:5: note: declared here
 int numa_node_to_cpus(int, struct bitmask *);
     ^
make[3]: *** [crafty.o] Error 1
make[3]: *** Waiting for unfinished jobs....
make[3]: Leaving directory `/home/louis/Documents/Chess/Crafty'
make[2]: *** [crafty-make] Error 2
make[2]: Leaving directory `/home/louis/Documents/Chess/Crafty'
make[1]: *** [unix-gcc-profile] Error 2
make[1]: Leaving directory `/home/louis/Documents/Chess/Crafty'
make: *** [profile] Error 2

[zullil]

First, since not all machines support NUMA, I have a -DNUMA Makefile option that turns this on. Leave -DNUMA off, and it doesn't do any NUMA-related tricks at all.

Including -DNUMA leads to the following error. This is on a linux system:

Code: Select all

$ make profile
make -j unix-gcc-profile
make[1]: Entering directory `/home/louis/Documents/Chess/Crafty'
make -j target=UNIX \
		CC=gcc-5 CXX=g++-5 \
		opt='-DTEST -DINLINEASM -DPOPCNT -DCPUS=20 -DAFFINITY -DNUMA' \
		CFLAGS='-Wall -Wno-array-bounds -pipe -O3 -march=native -fprofile-arcs \
		-pthread' \
		CXFLAGS='-Wall -Wno-array-bounds -pipe -O3 -march=native -fprofile-arcs \
		-pthread' \
		LDFLAGS=' -fprofile-arcs -pthread -lstdc++ ' \
		crafty-make
make[2]: Entering directory `/home/louis/Documents/Chess/Crafty'
make[3]: Entering directory `/home/louis/Documents/Chess/Crafty'
gcc-5 -Wall -Wno-array-bounds -pipe -O3 -march=native -fprofile-arcs \
-pthread -DTEST -DINLINEASM -DPOPCNT -DCPUS=20 -DAFFINITY -DNUMA -DUNIX -c crafty.c
g++-5 -c -Wall -Wno-array-bounds -pipe -O3 -march=native -fprofile-arcs \
-pthread -DTEST -DINLINEASM -DPOPCNT -DCPUS=20 -DAFFINITY -DNUMA -DUNIX egtb.cpp
In file included from crafty.c:45:0:
main.c: In function ‘main’:
main.c:4309:26: warning: passing argument 2 of ‘numa_node_to_cpus’ from incompatible pointer type [-Wincompatible-pointer-types]
     numa_node_to_cpus(0, cpus, 64);
                          ^
In file included from main.c:9:0,
                 from crafty.c:45:
/usr/include/numa.h:283:5: note: expected ‘struct bitmask *’ but argument is of type ‘long unsigned int *’
 int numa_node_to_cpus(int, struct bitmask *);
     ^
In file included from crafty.c:45:0:
main.c:4309:5: error: too many arguments to function ‘numa_node_to_cpus’
     numa_node_to_cpus(0, cpus, 64);
     ^
In file included from main.c:9:0,
                 from crafty.c:45:
/usr/include/numa.h:283:5: note: declared here
 int numa_node_to_cpus(int, struct bitmask *);
     ^
make[3]: *** [crafty.o] Error 1
make[3]: *** Waiting for unfinished jobs....
make[3]: Leaving directory `/home/louis/Documents/Chess/Crafty'
make[2]: *** [crafty-make] Error 2
make[2]: Leaving directory `/home/louis/Documents/Chess/Crafty'
make[1]: *** [unix-gcc-profile] Error 2
make[1]: Leaving directory `/home/louis/Documents/Chess/Crafty'
make: *** [profile] Error 2

As a temporary workaround, I simply commented out the offending code block, whose purpose was to print the numa-configuration of the system:

Code: Select all

/*
 ************************************************************
 *                                                          *
 *  If NUMA is enabled, report on the current machine       *
 *  configuration.                                          *
 *                                                          *
 ************************************************************
 
#if defined(NUMA) && defined(UNIX)
  if (numa_available() >= 0) {
    unsigned long cpus[8];

    numa_node_to_cpus(0, cpus, 64);
    Print(32, "\nMachine is NUMA, %d nodes (%d cpus/node)\n\n",
        numa_max_node() + 1, PopCnt(cpus[0]));
  }
#endif
*/

[abulmo]

Two different api versions exist. To use the old version try:
-DNUMA_VERSION1_COMPATIBILITY

[jdart]

This library looks possibly useful (it is BSD-licensed):

http://www.open-mpi.org/projects/hwloc/

--Jon

[zullil]

Two different api versions exist. To use the old version try:
-DNUMA_VERSION1_COMPATIBILITY

Tried that first, before commenting out the code. Led to

Code: Select all

gcc-5 -Wall -Wno-array-bounds -pipe -O3 -march=native -fprofile-arcs \
-pthread -DTEST -DINLINEASM -DPOPCNT -DCPUS=20 -DAFFINITY -DNUMA_VERSION1_COMPATIBILITY -DNUMA -DUNIX -c crafty.c
g++-5 -c -Wall -Wno-array-bounds -pipe -O3 -march=native -fprofile-arcs \
-pthread -DTEST -DINLINEASM -DPOPCNT -DCPUS=20 -DAFFINITY -DNUMA_VERSION1_COMPATIBILITY -DNUMA -DUNIX egtb.cpp
gcc-5 -fprofile-arcs -pthread -lstdc++  -o crafty crafty.o egtb.o -lm  
crafty.o: In function `main':
crafty.c:(.text.startup+0x451): undefined reference to `numa_available'
crafty.c:(.text.startup+0x66b): undefined reference to `numa_node_to_cpus'
crafty.c:(.text.startup+0x68a): undefined reference to `numa_max_node'
collect2: error: ld returned 1 exit status
make[3]: *** [crafty] Error 1
make[3]: Leaving directory `/home/louis/Documents/Chess/Crafty'
make[2]: *** [crafty-make] Error 2
make[2]: Leaving directory `/home/louis/Documents/Chess/Crafty'
make[1]: *** [unix-gcc-profile] Error 2
make[1]: Leaving directory `/home/louis/Documents/Chess/Crafty'
make: *** [profile] Error 2

[bob]

This library looks possibly useful (it is BSD-licensed):

http://www.open-mpi.org/projects/hwloc/

--Jon

Yes it does. I actually started writing something like that using the basic CPUID instruction. But it quickly became so messy I decided it was not worth it. I will poke around in that when I have time.

[bob]

First, since not all machines support NUMA, I have a -DNUMA Makefile option that turns this on. Leave -DNUMA off, and it doesn't do any NUMA-related tricks at all.

Including -DNUMA leads to the following error. This is on a linux system:

Code: Select all

$ make profile
make -j unix-gcc-profile
make[1]: Entering directory `/home/louis/Documents/Chess/Crafty'
make -j target=UNIX \
		CC=gcc-5 CXX=g++-5 \
		opt='-DTEST -DINLINEASM -DPOPCNT -DCPUS=20 -DAFFINITY -DNUMA' \
		CFLAGS='-Wall -Wno-array-bounds -pipe -O3 -march=native -fprofile-arcs \
		-pthread' \
		CXFLAGS='-Wall -Wno-array-bounds -pipe -O3 -march=native -fprofile-arcs \
		-pthread' \
		LDFLAGS=' -fprofile-arcs -pthread -lstdc++ ' \
		crafty-make
make[2]: Entering directory `/home/louis/Documents/Chess/Crafty'
make[3]: Entering directory `/home/louis/Documents/Chess/Crafty'
gcc-5 -Wall -Wno-array-bounds -pipe -O3 -march=native -fprofile-arcs \
-pthread -DTEST -DINLINEASM -DPOPCNT -DCPUS=20 -DAFFINITY -DNUMA -DUNIX -c crafty.c
g++-5 -c -Wall -Wno-array-bounds -pipe -O3 -march=native -fprofile-arcs \
-pthread -DTEST -DINLINEASM -DPOPCNT -DCPUS=20 -DAFFINITY -DNUMA -DUNIX egtb.cpp
In file included from crafty.c:45:0:
main.c: In function ‘main’:
main.c:4309:26: warning: passing argument 2 of ‘numa_node_to_cpus’ from incompatible pointer type [-Wincompatible-pointer-types]
     numa_node_to_cpus(0, cpus, 64);
                          ^
In file included from main.c:9:0,
                 from crafty.c:45:
/usr/include/numa.h:283:5: note: expected ‘struct bitmask *’ but argument is of type ‘long unsigned int *’
 int numa_node_to_cpus(int, struct bitmask *);
     ^
In file included from crafty.c:45:0:
main.c:4309:5: error: too many arguments to function ‘numa_node_to_cpus’
     numa_node_to_cpus(0, cpus, 64);
     ^
In file included from main.c:9:0,
                 from crafty.c:45:
/usr/include/numa.h:283:5: note: declared here
 int numa_node_to_cpus(int, struct bitmask *);
     ^
make[3]: *** [crafty.o] Error 1
make[3]: *** Waiting for unfinished jobs....
make[3]: Leaving directory `/home/louis/Documents/Chess/Crafty'
make[2]: *** [crafty-make] Error 2
make[2]: Leaving directory `/home/louis/Documents/Chess/Crafty'
make[1]: *** [unix-gcc-profile] Error 2
make[1]: Leaving directory `/home/louis/Documents/Chess/Crafty'
make: *** [profile] Error 2

That library call was changed. I don't really use any of those any longer, that one was used strictly to print the greeting noting that this is a NUMA box.

Is this the latest 25.0 or the one I sent you a while back. I did fix that to use a different numa library routine, and thought it was in 25.0 as released?

[bob]

Two different api versions exist. To use the old version try:
-DNUMA_VERSION1_COMPATIBILITY

Tried that first, before commenting out the code. Led to

Code: Select all

gcc-5 -Wall -Wno-array-bounds -pipe -O3 -march=native -fprofile-arcs \
-pthread -DTEST -DINLINEASM -DPOPCNT -DCPUS=20 -DAFFINITY -DNUMA_VERSION1_COMPATIBILITY -DNUMA -DUNIX -c crafty.c
g++-5 -c -Wall -Wno-array-bounds -pipe -O3 -march=native -fprofile-arcs \
-pthread -DTEST -DINLINEASM -DPOPCNT -DCPUS=20 -DAFFINITY -DNUMA_VERSION1_COMPATIBILITY -DNUMA -DUNIX egtb.cpp
gcc-5 -fprofile-arcs -pthread -lstdc++  -o crafty crafty.o egtb.o -lm  
crafty.o: In function `main':
crafty.c:(.text.startup+0x451): undefined reference to `numa_available'
crafty.c:(.text.startup+0x66b): undefined reference to `numa_node_to_cpus'
crafty.c:(.text.startup+0x68a): undefined reference to `numa_max_node'
collect2: error: ld returned 1 exit status
make[3]: *** [crafty] Error 1
make[3]: Leaving directory `/home/louis/Documents/Chess/Crafty'
make[2]: *** [crafty-make] Error 2
make[2]: Leaving directory `/home/louis/Documents/Chess/Crafty'
make[1]: *** [unix-gcc-profile] Error 2
make[1]: Leaving directory `/home/louis/Documents/Chess/Crafty'
make: *** [profile] Error 2

I will release 25.1 pretty soon. It has some additional NUMA cleanup done. I was not quite ready with it when we released 25.0, but this is the version that distributes the various hash tables across NUMA nodes so that each node will have about the same fraction of each hash table resident locally to avoid hot-spots.

[zullil]

First, since not all machines support NUMA, I have a -DNUMA Makefile option that turns this on. Leave -DNUMA off, and it doesn't do any NUMA-related tricks at all.

Including -DNUMA leads to the following error. This is on a linux system:

Code: Select all

$ make profile
make -j unix-gcc-profile
make[1]: Entering directory `/home/louis/Documents/Chess/Crafty'
make -j target=UNIX \
		CC=gcc-5 CXX=g++-5 \
		opt='-DTEST -DINLINEASM -DPOPCNT -DCPUS=20 -DAFFINITY -DNUMA' \
		CFLAGS='-Wall -Wno-array-bounds -pipe -O3 -march=native -fprofile-arcs \
		-pthread' \
		CXFLAGS='-Wall -Wno-array-bounds -pipe -O3 -march=native -fprofile-arcs \
		-pthread' \
		LDFLAGS=' -fprofile-arcs -pthread -lstdc++ ' \
		crafty-make
make[2]: Entering directory `/home/louis/Documents/Chess/Crafty'
make[3]: Entering directory `/home/louis/Documents/Chess/Crafty'
gcc-5 -Wall -Wno-array-bounds -pipe -O3 -march=native -fprofile-arcs \
-pthread -DTEST -DINLINEASM -DPOPCNT -DCPUS=20 -DAFFINITY -DNUMA -DUNIX -c crafty.c
g++-5 -c -Wall -Wno-array-bounds -pipe -O3 -march=native -fprofile-arcs \
-pthread -DTEST -DINLINEASM -DPOPCNT -DCPUS=20 -DAFFINITY -DNUMA -DUNIX egtb.cpp
In file included from crafty.c:45:0:
main.c: In function ‘main’:
main.c:4309:26: warning: passing argument 2 of ‘numa_node_to_cpus’ from incompatible pointer type [-Wincompatible-pointer-types]
     numa_node_to_cpus(0, cpus, 64);
                          ^
In file included from main.c:9:0,
                 from crafty.c:45:
/usr/include/numa.h:283:5: note: expected ‘struct bitmask *’ but argument is of type ‘long unsigned int *’
 int numa_node_to_cpus(int, struct bitmask *);
     ^
In file included from crafty.c:45:0:
main.c:4309:5: error: too many arguments to function ‘numa_node_to_cpus’
     numa_node_to_cpus(0, cpus, 64);
     ^
In file included from main.c:9:0,
                 from crafty.c:45:
/usr/include/numa.h:283:5: note: declared here
 int numa_node_to_cpus(int, struct bitmask *);
     ^
make[3]: *** [crafty.o] Error 1
make[3]: *** Waiting for unfinished jobs....
make[3]: Leaving directory `/home/louis/Documents/Chess/Crafty'
make[2]: *** [crafty-make] Error 2
make[2]: Leaving directory `/home/louis/Documents/Chess/Crafty'
make[1]: *** [unix-gcc-profile] Error 2
make[1]: Leaving directory `/home/louis/Documents/Chess/Crafty'
make: *** [profile] Error 2

That library call was changed. I don't really use any of those any longer, that one was used strictly to print the greeting noting that this is a NUMA box.

Is this the latest 25.0 or the one I sent you a while back. I did fix that to use a different numa library routine, and thought it was in 25.0 as released?

Bob, the code that causes this error is in the released 25.0.