Stockfish Cluster MPI

Discussion of anything and everything relating to chess playing software and machines.

Moderators: hgm, Harvey Williamson, bob

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
fastgm
Posts: 391
Joined: Mon Aug 19, 2013 4:57 pm
Contact:

Stockfish Cluster MPI

Post by fastgm » Sun Feb 03, 2019 7:22 am

I would like to test the stockfish cluster version. https://github.com/official-stockfish/S ... ee/cluster
Can someone provide an .exe file?

komodoslash001
Posts: 8
Joined: Fri Feb 01, 2019 10:56 am
Full name: Jeremy Tyrania

Re: Stockfish Cluster MPI

Post by komodoslash001 » Sun Feb 03, 2019 7:48 am

I compiled it but no idea how it works. :P
https://www.dropbox.com/s/x2an0hkj1r554 ... r.zip?dl=0

fastgm
Posts: 391
Joined: Mon Aug 19, 2013 4:57 pm
Contact:

Re: Stockfish Cluster MPI

Post by fastgm » Sun Feb 03, 2019 10:43 am

Thank you for your effort, but unfortunately it does not work.

Did you use the compiler options in the readme file?

To build the cluster branch, it is sufficient to specify COMPILER=mpicxx on the make command line, and do a clean build:

make -j ARCH=x86-64-modern clean build COMPILER=mpicxx
If the name of the compiler wrapper (typically mpicxx, but sometimes e.g. CC) does not match mpi an edit to the Makefile is required. Make sure that the MPI installation is configured to support MPI_THREAD_MULTIPLE, this might require adding system specific compiler options to the Makefile. Stockfish employs non-blocking (asynchronous) communication, and benefits from an MPI implementation that efficiently supports this. Some MPI implentations might benefit from leaving 1 core/thread free for these asynchronous communications, and might require setting additional environment variables. Refer to your MPI documentation for more info.

komodoslash001
Posts: 8
Joined: Fri Feb 01, 2019 10:56 am
Full name: Jeremy Tyrania

Re: Stockfish Cluster MPI

Post by komodoslash001 » Sun Feb 03, 2019 4:39 pm

Ahh I tried to compile it like normal stockfish :p I turned on MPI in the makefile and also checked during the compile it said mpi: 'yes'. Hope this one works. :)

https://www.dropbox.com/s/ejmlrevz8nuzk ... r.zip?dl=0

Hugo
Posts: 775
Joined: Tue Dec 01, 2009 10:10 am

Re: Stockfish Cluster MPI

Post by Hugo » Sun Feb 03, 2019 10:01 pm

Question:
Why does it start like a normal engine?
should this engine start in consolemode only if MPI is installed ?

regards, C.K.

komodoslash001 wrote:
Sun Feb 03, 2019 4:39 pm
Ahh I tried to compile it like normal stockfish :p I turned on MPI in the makefile and also checked during the compile it said mpi: 'yes'. Hope this one works. :)

https://www.dropbox.com/s/ejmlrevz8nuzk ... r.zip?dl=0

TimoK
Posts: 97
Joined: Sun Jan 03, 2010 11:28 am
Location: Hamburg

Re: Stockfish Cluster MPI

Post by TimoK » Mon Feb 04, 2019 6:04 am

First of all: Thanks to komodoslash001 for compiling!

Unfortunately it doesn't work. If you start it with someting like: "mpiexec -hosts 2 host1 host2 /path/to/stockfish-executable" then Stockfish starts on both hosts but the two processes don't communicate with each other. Maybe there is still something missing in the compile or it does only work when compiling for Linux.

Best regards
Timo

Werewolf
Posts: 1175
Joined: Thu Sep 18, 2008 8:24 pm

Re: Stockfish Cluster MPI

Post by Werewolf » Mon Feb 04, 2019 7:47 am

This is such an exciting project, it'd be great to get it working easily.

I really hope the SF team make this easily accessible with clear instructions of how to link multiple PCs. It could kick start new interest in alpa-betas.

petero2
Posts: 576
Joined: Mon Apr 19, 2010 5:07 pm
Location: Sweden
Contact:

Re: Stockfish Cluster MPI

Post by petero2 » Fri Feb 08, 2019 5:44 pm

fastgm wrote:
Sun Feb 03, 2019 7:22 am
I would like to test the stockfish cluster version. https://github.com/official-stockfish/S ... ee/cluster
Can someone provide an .exe file?
I have uploaded stockfish_cluster_win64.7z to this public dropbox folder:

https://www.dropbox.com/sh/v16dye7ww33x ... Oxasa/test

Stockfish is compiled for the x86-64-modern architecture. There are some limitations though:
  • Windows 8 or later is needed to support MPI_THREAD_MULTIPLE which stockfish is using.
  • Version 10.0 of MS-MPI is needed for bug fixes in the MPI_THREAD_MULTIPLE handling. You can get it from here: https://www.microsoft.com/en-us/downloa ... x?id=57467
  • I had to disable link time optimization (-flto) when compiling stockfish, otherwise the program crashes. Maybe this happens because I cross compile from linux to windows.
You can run this engine basically the same way as you run the cluster version of texel. Here is an example adjusted from the texel documentation:

If there are two computers called host1 and host2 and MS-MPI is installed on both computers, proceed as follows:
  1. On all computers, log in as the same user.
  2. On all computers, add firewall exceptions to allow the programs mpiexec and smpd (located in C:\Program Files\Microsoft MPI\Bin) to communicate over the network.
  3. On all computers, start a command prompt and execute: smpd -d 0
  4. Make sure stockfish is installed in the same directory on all computers.
  5. On the host1 computer, start a command prompt and execute:
    cd /directory/where/stockfish/is/installed
    mpiexec -hosts 2 host1 host2 stockfish-cluster.exe
These posts might help too: It should also be possible to use the InBetween program instead of the "runcmd.exe" program mentioned in the last post.

henk2
Posts: 30
Joined: Mon Jan 14, 2019 6:55 am
Full name: Henk Verbaasdonk

Re: Stockfish Cluster MPI

Post by henk2 » Fri Feb 08, 2019 6:46 pm

fastgm wrote:
Sun Feb 03, 2019 7:22 am
I would like to test the stockfish cluster version. https://github.com/official-stockfish/S ... ee/cluster
Can someone provide an .exe file?
Are you going to let two Clusterfishes battle it out until depth 70?

Geonerd
Posts: 62
Joined: Fri Mar 10, 2017 12:44 am

Re: Stockfish Cluster MPI

Post by Geonerd » Fri Feb 08, 2019 7:25 pm

petero2 wrote:
Fri Feb 08, 2019 5:44 pm

I have uploaded stockfish_cluster_win64.7z to this public dropbox folder:
Thank you for the compile and specific instructions! :D

Post Reply