How do you build a chess cluster?

[Uri]

Since I don't have a lot of money, 4 years from now when I'll buy a more powerful computer, I may want to connect it to the old ones.

The question is how do I do this difficult task? How do you build a computer chess cluster and what software can manage this?

M ANSARI, an expert in computer chess said in a different forum that there is a software called Chessnet which allows you to name seperate UCI engines and allocate an IP address to each engine. Unfortunately, I searched the net and found no clustering software called Chessnet.

So could you also recommend me some programs which do this?

[bob]

Since I don't have a lot of money, 4 years from now when I'll buy a more powerful computer, I may want to connect it to the old ones.

The question is how do I do this difficult task? How do you build a computer chess cluster and what software can manage this?

M ANSARI, an expert in computer chess said in a different forum that there is a software called Chessnet which allows you to name seperate UCI engines and allocate an IP address to each engine. Unfortunately, I searched the net and found no clustering software called Chessnet.

So could you also recommend me some programs which do this?

1. A cluster is simply 2 or more nodes (computers) connected via some sort of local LAN. This LAN can be as simple as a 100mbit/sec ethernet, or as sophisticated as a 40gbit/second infiniband connection which is pretty pricey.

2. Software to use the cluster is more complex because I am not aware of any current program that will use a cluster. GCP has said the next version of Sjeng will do so. A future version of Crafty (hopefully this year) will also do so. But right now the pickin's are slim to none. You don't need any special software unless the engine author chooses to use something like MPI or openMP or whatever rather than straight TCP/IP to communicate between nodes...

[Uri]

Software to use the cluster is more complex because I am not aware of any current program that will use a cluster. GCP has said the next version of Sjeng will do so. A future version of Crafty (hopefully this year) will also do so. But right now the pickin's are slim to none. You don't need any special software unless the engine author chooses to use something like MPI or openMP or whatever rather than straight TCP/IP to communicate between nodes...

So what programs do I need (besides Deep Rybka 3) to build a Rybka cluster? Could you recommend me some?

[Zach Wegner]

A future version of Crafty (hopefully this year) will also do so.

How's this going, by the way? Got anything running yet?

[M ANSARI]

I think you misunderstood me, I did not say that Chessnet is a clustering software, I said that it can be used as a way to cluster different computers together. Chessnet has already been used for some time in "manual" clusters ... mainly in freestyle events. Basically you do the following:

You install the Chessnet client for 5 computers and then you assign each computer with an engine running an IP address. So computer 1 would be say engine_1 and computer 2 engine_2 etc... Now from your master computer you can load each individual engine as an additional engine and thus on the master computer you would have the output of the master plus output of the 4 loaded engines which you have loaded from the GUI. Basically what you have done is have the output of 5 computers on 1 computer. Now the problem is that obviously if each is running an identical engine, then you would have the same engine output ... so the trick is to have each engine check the next relevant line in the search. In freestyle this is done manually by having the master look at multiple PV's then you input each one of the PV's for each of the loaded engines to ponder. Ofcourse in freestyle you can have several different engines running, but the human then would have to be very good at parsing the different evaluations and thus avoiding false positives in evaluations. That is why if you want to automate this in a real automatic cluster using automated clustering software, you would do best to have identical hardware and an identical engine. Not having that could pollute your output.

Obviously there is more to that and Vas and others must have found ways to improve of this primitve cluster setup, but I guess you could write a driver or some script that could automate such a setup, and it would not necessarily have to be engine specific.

By the way the netChess software described is on this site

http://home.arcor.de/bernhard.wallner/netChess.html

[Uri]

I think you misunderstood me, I did not say that Chessnet is a clustering software, I said that it can be used as a way to cluster different computers together. Chessnet has already been used for some time in "manual" clusters ... mainly in freestyle events. Basically you do the following:

You install the Chessnet client for 5 computers and then you assign each computer with an engine running an IP address. So computer 1 would be say engine_1 and computer 2 engine_2 etc... Now from your master computer you can load each individual engine as an additional engine and thus on the master computer you would have the output of the master plus output of the 4 loaded engines which you have loaded from the GUI. Basically what you have done is have the output of 5 computers on 1 computer. Now the problem is that obviously if each is running an identical engine, then you would have the same engine output ... so the trick is to have each engine check the next relevant line in the search. In freestyle this is done manually by having the master look at multiple PV's then you input each one of the PV's for each of the loaded engines to ponder. Ofcourse in freestyle you can have several different engines running, but the human then would have to be very good at parsing the different evaluations and thus avoiding false positives in evaluations. That is why if you want to automate this in a real automatic cluster using automated clustering software, you would do best to have identical hardware and an identical engine. Not having that could pollute your output.

Obviously there is more to that and Vas and others must have found ways to improve of this primitve cluster setup, but I guess you could write a driver or some script that could automate such a setup, and it would not necessarily have to be engine specific.

By the way the netChess software described is on this site

http://home.arcor.de/bernhard.wallner/netChess.html

And where can I find Chessnet/netChess or however it's called?

[AdminX]

I think you misunderstood me, I did not say that Chessnet is a clustering software, I said that it can be used as a way to cluster different computers together. Chessnet has already been used for some time in "manual" clusters ... mainly in freestyle events. Basically you do the following:

You install the Chessnet client for 5 computers and then you assign each computer with an engine running an IP address. So computer 1 would be say engine_1 and computer 2 engine_2 etc... Now from your master computer you can load each individual engine as an additional engine and thus on the master computer you would have the output of the master plus output of the 4 loaded engines which you have loaded from the GUI. Basically what you have done is have the output of 5 computers on 1 computer. Now the problem is that obviously if each is running an identical engine, then you would have the same engine output ... so the trick is to have each engine check the next relevant line in the search. In freestyle this is done manually by having the master look at multiple PV's then you input each one of the PV's for each of the loaded engines to ponder. Ofcourse in freestyle you can have several different engines running, but the human then would have to be very good at parsing the different evaluations and thus avoiding false positives in evaluations. That is why if you want to automate this in a real automatic cluster using automated clustering software, you would do best to have identical hardware and an identical engine. Not having that could pollute your output.

Obviously there is more to that and Vas and others must have found ways to improve of this primitve cluster setup, but I guess you could write a driver or some script that could automate such a setup, and it would not necessarily have to be engine specific.

By the way the netChess software described is on this site

http://home.arcor.de/bernhard.wallner/netChess.html

And where can I find Chessnet/netChess or however it's called?

It's right there in his link.

[bob]

Software to use the cluster is more complex because I am not aware of any current program that will use a cluster. GCP has said the next version of Sjeng will do so. A future version of Crafty (hopefully this year) will also do so. But right now the pickin's are slim to none. You don't need any special software unless the engine author chooses to use something like MPI or openMP or whatever rather than straight TCP/IP to communicate between nodes...

So what programs do I need (besides Deep Rybka 3) to build a Rybka cluster? Could you recommend me some?

I do not know what Rybka uses for cluster communication, so I have no way of responding. It could use bare TCP/IP which means you just need machines with network connections and a network switch to interconnect them. It might require additional software such as mpich or openmp, I do not know what was used.

[bob]

A future version of Crafty (hopefully this year) will also do so.

How's this going, by the way? Got anything running yet?

No. But got lots of the design and some of the coding done... Every time I get started on details, I discover a better approach to do something and take about 10 steps back for every step forward.

[Gian-Carlo Pascutto]

So what programs do I need (besides Deep Rybka 3) to build a Rybka cluster? Could you recommend me some?

Rybka 3 does not support clustering either. All the programs that do are in development and unreleased to the public.

I guess near the end of the year we might see cluster versions of Rybka 4, Crafty 24 and Deep Sjeng 4 becoming available publicly...