How do you build a chess cluster?

Discussion of anything and everything relating to chess playing software and machines.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
Uri
Posts: 418
Joined: Thu Dec 27, 2007 8:34 pm

How do you build a chess cluster?

Post by Uri » Thu May 14, 2009 9:55 pm

Since I don't have a lot of money, 4 years from now when I'll buy a more powerful computer, I may want to connect it to the old ones.

The question is how do I do this difficult task? How do you build a computer chess cluster and what software can manage this?

M ANSARI, an expert in computer chess said in a different forum that there is a software called Chessnet which allows you to name seperate UCI engines and allocate an IP address to each engine. Unfortunately, I searched the net and found no clustering software called Chessnet.

So could you also recommend me some programs which do this?

bob
Posts: 20478
Joined: Mon Feb 27, 2006 6:30 pm
Location: Birmingham, AL

Re: How do you build a chess cluster?

Post by bob » Thu May 14, 2009 10:19 pm

Uri wrote:Since I don't have a lot of money, 4 years from now when I'll buy a more powerful computer, I may want to connect it to the old ones.

The question is how do I do this difficult task? How do you build a computer chess cluster and what software can manage this?

M ANSARI, an expert in computer chess said in a different forum that there is a software called Chessnet which allows you to name seperate UCI engines and allocate an IP address to each engine. Unfortunately, I searched the net and found no clustering software called Chessnet.

So could you also recommend me some programs which do this?
1. A cluster is simply 2 or more nodes (computers) connected via some sort of local LAN. This LAN can be as simple as a 100mbit/sec ethernet, or as sophisticated as a 40gbit/second infiniband connection which is pretty pricey.

2. Software to use the cluster is more complex because I am not aware of any current program that will use a cluster. GCP has said the next version of Sjeng will do so. A future version of Crafty (hopefully this year) will also do so. But right now the pickin's are slim to none. You don't need any special software unless the engine author chooses to use something like MPI or openMP or whatever rather than straight TCP/IP to communicate between nodes...

Uri
Posts: 418
Joined: Thu Dec 27, 2007 8:34 pm

Re: How do you build a chess cluster?

Post by Uri » Thu May 14, 2009 10:34 pm

bob wrote:Software to use the cluster is more complex because I am not aware of any current program that will use a cluster. GCP has said the next version of Sjeng will do so. A future version of Crafty (hopefully this year) will also do so. But right now the pickin's are slim to none. You don't need any special software unless the engine author chooses to use something like MPI or openMP or whatever rather than straight TCP/IP to communicate between nodes...
So what programs do I need (besides Deep Rybka 3) to build a Rybka cluster? Could you recommend me some?

User avatar
Zach Wegner
Posts: 1922
Joined: Wed Mar 08, 2006 11:51 pm
Location: Earth
Contact:

Re: How do you build a chess cluster?

Post by Zach Wegner » Thu May 14, 2009 10:44 pm

bob wrote:A future version of Crafty (hopefully this year) will also do so.
How's this going, by the way? Got anything running yet?

User avatar
M ANSARI
Posts: 3408
Joined: Thu Mar 16, 2006 6:10 pm

Re: How do you build a chess cluster?

Post by M ANSARI » Fri May 15, 2009 1:37 am

I think you misunderstood me, I did not say that Chessnet is a clustering software, I said that it can be used as a way to cluster different computers together. Chessnet has already been used for some time in "manual" clusters ... mainly in freestyle events. Basically you do the following:

You install the Chessnet client for 5 computers and then you assign each computer with an engine running an IP address. So computer 1 would be say engine_1 and computer 2 engine_2 etc... Now from your master computer you can load each individual engine as an additional engine and thus on the master computer you would have the output of the master plus output of the 4 loaded engines which you have loaded from the GUI. Basically what you have done is have the output of 5 computers on 1 computer. Now the problem is that obviously if each is running an identical engine, then you would have the same engine output ... so the trick is to have each engine check the next relevant line in the search. In freestyle this is done manually by having the master look at multiple PV's then you input each one of the PV's for each of the loaded engines to ponder. Ofcourse in freestyle you can have several different engines running, but the human then would have to be very good at parsing the different evaluations and thus avoiding false positives in evaluations. That is why if you want to automate this in a real automatic cluster using automated clustering software, you would do best to have identical hardware and an identical engine. Not having that could pollute your output.

Obviously there is more to that and Vas and others must have found ways to improve of this primitve cluster setup, but I guess you could write a driver or some script that could automate such a setup, and it would not necessarily have to be engine specific.

By the way the netChess software described is on this site

http://home.arcor.de/bernhard.wallner/netChess.html

Uri
Posts: 418
Joined: Thu Dec 27, 2007 8:34 pm

Re: How do you build a chess cluster?

Post by Uri » Fri May 15, 2009 2:06 am

M ANSARI wrote:I think you misunderstood me, I did not say that Chessnet is a clustering software, I said that it can be used as a way to cluster different computers together. Chessnet has already been used for some time in "manual" clusters ... mainly in freestyle events. Basically you do the following:

You install the Chessnet client for 5 computers and then you assign each computer with an engine running an IP address. So computer 1 would be say engine_1 and computer 2 engine_2 etc... Now from your master computer you can load each individual engine as an additional engine and thus on the master computer you would have the output of the master plus output of the 4 loaded engines which you have loaded from the GUI. Basically what you have done is have the output of 5 computers on 1 computer. Now the problem is that obviously if each is running an identical engine, then you would have the same engine output ... so the trick is to have each engine check the next relevant line in the search. In freestyle this is done manually by having the master look at multiple PV's then you input each one of the PV's for each of the loaded engines to ponder. Ofcourse in freestyle you can have several different engines running, but the human then would have to be very good at parsing the different evaluations and thus avoiding false positives in evaluations. That is why if you want to automate this in a real automatic cluster using automated clustering software, you would do best to have identical hardware and an identical engine. Not having that could pollute your output.

Obviously there is more to that and Vas and others must have found ways to improve of this primitve cluster setup, but I guess you could write a driver or some script that could automate such a setup, and it would not necessarily have to be engine specific.

By the way the netChess software described is on this site

http://home.arcor.de/bernhard.wallner/netChess.html
And where can I find Chessnet/netChess or however it's called?

User avatar
AdminX
Posts: 5133
Joined: Mon Mar 13, 2006 1:34 pm
Location: Acworth, GA
Contact:

Re: How do you build a chess cluster?

Post by AdminX » Fri May 15, 2009 2:09 am

Uri wrote:
M ANSARI wrote:I think you misunderstood me, I did not say that Chessnet is a clustering software, I said that it can be used as a way to cluster different computers together. Chessnet has already been used for some time in "manual" clusters ... mainly in freestyle events. Basically you do the following:

You install the Chessnet client for 5 computers and then you assign each computer with an engine running an IP address. So computer 1 would be say engine_1 and computer 2 engine_2 etc... Now from your master computer you can load each individual engine as an additional engine and thus on the master computer you would have the output of the master plus output of the 4 loaded engines which you have loaded from the GUI. Basically what you have done is have the output of 5 computers on 1 computer. Now the problem is that obviously if each is running an identical engine, then you would have the same engine output ... so the trick is to have each engine check the next relevant line in the search. In freestyle this is done manually by having the master look at multiple PV's then you input each one of the PV's for each of the loaded engines to ponder. Ofcourse in freestyle you can have several different engines running, but the human then would have to be very good at parsing the different evaluations and thus avoiding false positives in evaluations. That is why if you want to automate this in a real automatic cluster using automated clustering software, you would do best to have identical hardware and an identical engine. Not having that could pollute your output.

Obviously there is more to that and Vas and others must have found ways to improve of this primitve cluster setup, but I guess you could write a driver or some script that could automate such a setup, and it would not necessarily have to be engine specific.

By the way the netChess software described is on this site

http://home.arcor.de/bernhard.wallner/netChess.html
And where can I find Chessnet/netChess or however it's called?
It's right there in his link.
"Good decisions come from experience, and experience comes from bad decisions."
__________________________________________________________________
Ted Summers

bob
Posts: 20478
Joined: Mon Feb 27, 2006 6:30 pm
Location: Birmingham, AL

Re: How do you build a chess cluster?

Post by bob » Fri May 15, 2009 4:58 am

Uri wrote:
bob wrote:Software to use the cluster is more complex because I am not aware of any current program that will use a cluster. GCP has said the next version of Sjeng will do so. A future version of Crafty (hopefully this year) will also do so. But right now the pickin's are slim to none. You don't need any special software unless the engine author chooses to use something like MPI or openMP or whatever rather than straight TCP/IP to communicate between nodes...
So what programs do I need (besides Deep Rybka 3) to build a Rybka cluster? Could you recommend me some?
I do not know what Rybka uses for cluster communication, so I have no way of responding. It could use bare TCP/IP which means you just need machines with network connections and a network switch to interconnect them. It might require additional software such as mpich or openmp, I do not know what was used.

bob
Posts: 20478
Joined: Mon Feb 27, 2006 6:30 pm
Location: Birmingham, AL

Re: How do you build a chess cluster?

Post by bob » Fri May 15, 2009 4:59 am

Zach Wegner wrote:
bob wrote:A future version of Crafty (hopefully this year) will also do so.
How's this going, by the way? Got anything running yet?
No. But got lots of the design and some of the coding done... Every time I get started on details, I discover a better approach to do something and take about 10 steps back for every step forward.

Gian-Carlo Pascutto
Posts: 1166
Joined: Sat Dec 13, 2008 6:00 pm
Contact:

Re: How do you build a chess cluster?

Post by Gian-Carlo Pascutto » Fri May 15, 2009 6:21 am

Uri wrote: So what programs do I need (besides Deep Rybka 3) to build a Rybka cluster? Could you recommend me some?
Rybka 3 does not support clustering either. All the programs that do are in development and unreleased to the public.

I guess near the end of the year we might see cluster versions of Rybka 4, Crafty 24 and Deep Sjeng 4 becoming available publicly...

Post Reply