Basic automated testing

Discussion of chess software programming and technical issues.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
mar
Posts: 2010
Joined: Fri Nov 26, 2010 1:00 pm
Location: Czech Republic
Full name: Martin Sedlak

Re: Basic automated testing

Post by mar » Mon Oct 01, 2018 1:37 pm

AndrewGrant wrote:
Sun Sep 30, 2018 10:51 pm
My instance is running here http://chess.grantnet.us/index/ , if you want to get an idea of the work flow.

You would be able to connect any of your machines, and the results will be collected together.

Time controls will be scaled per CPU, IE in the case that one machine is faster or something of the sort.

In case its not clear, I wrote OpenBench
That looks really nice. How do you supply test binaries? It seems you use branches in git so I assume it just fetches the branches and rebuilds on client?
Martin Sedlak

AndrewGrant
Posts: 494
Joined: Tue Apr 19, 2016 4:08 am
Location: U.S.A
Full name: Andrew Grant
Contact:

Re: Basic automated testing

Post by AndrewGrant » Mon Oct 01, 2018 6:13 pm

mar wrote:
Mon Oct 01, 2018 1:37 pm
AndrewGrant wrote:
Sun Sep 30, 2018 10:51 pm
My instance is running here http://chess.grantnet.us/index/ , if you want to get an idea of the work flow.

You would be able to connect any of your machines, and the results will be collected together.

Time controls will be scaled per CPU, IE in the case that one machine is faster or something of the sort.

In case its not clear, I wrote OpenBench
That looks really nice. How do you supply test binaries? It seems you use branches in git so I assume it just fetches the branches and rebuilds on client?
Correct. OpenBench assumes that gcc/g++ and make are on the system path. It will pull down a zip file from github -- no actual git comments are taking place.

AndrewGrant
Posts: 494
Joined: Tue Apr 19, 2016 4:08 am
Location: U.S.A
Full name: Andrew Grant
Contact:

Re: Basic automated testing

Post by AndrewGrant » Mon Oct 01, 2018 6:22 pm

cdani wrote:
Mon Oct 01, 2018 3:53 am
AndrewGrant wrote:
Sun Sep 30, 2018 10:51 pm
mar wrote:
Sun Sep 30, 2018 8:40 pm
Ratosh wrote:
Fri Sep 28, 2018 6:31 pm
I'd recommend OpenBench.
Interesting, what does this OpenBench do exactly? It seems like a wrapper on top of cutechess-cli with some client-server functionality.
If it allows me say to test on say two quads at the same time + gather the results then this would cut my testing time in half,
which sounds really interesting.
My instance is running here http://chess.grantnet.us/index/ , if you want to get an idea of the work flow.

You would be able to connect any of your machines, and the results will be collected together.

Time controls will be scaled per CPU, IE in the case that one machine is faster or something of the sort.

In case its not clear, I wrote OpenBench
I knew you have it, but I find is good time to congratulate you for your nice and interesting work! :-)
Thanks. The same can be said for Andscacs :)

jdart
Posts: 3835
Joined: Fri Mar 10, 2006 4:23 am
Location: http://www.arasanchess.org

Re: Basic automated testing

Post by jdart » Tue Oct 02, 2018 10:27 pm

What do your cutechess-cli scripts do? This might be what I am looking for (OpenBench looks too advanced for my current needs)
They are pretty simple. They distribute a compiled binary to a set of directories on a set of networked machines. Then they run a bunch of parallel matches on each machine. Then they gather all the completed games into a big game file and run BayesELO on it.

Unfortunately I have not bothered to make them very portable or reusable, so they have some hardcoded machine names, paths, etc.

--Jon

Post Reply