Page 1 of 2

Re: Basic automated testing

Posted: Sun Sep 30, 2018 10:51 pm
by AndrewGrant
mar wrote:
Sun Sep 30, 2018 8:40 pm
Ratosh wrote:
Fri Sep 28, 2018 6:31 pm
I'd recommend OpenBench.
Interesting, what does this OpenBench do exactly? It seems like a wrapper on top of cutechess-cli with some client-server functionality.
If it allows me say to test on say two quads at the same time + gather the results then this would cut my testing time in half,
which sounds really interesting.
My instance is running here http://chess.grantnet.us/index/ , if you want to get an idea of the work flow.

You would be able to connect any of your machines, and the results will be collected together.

Time controls will be scaled per CPU, IE in the case that one machine is faster or something of the sort.

In case its not clear, I wrote OpenBench

Re: Basic automated testing

Posted: Mon Oct 01, 2018 3:21 am
by jdart
I have a bunch of unit tests that include basic functionality. One of the components is perft (https://www.chessprogramming.org/Perft), which is quite useful for sanity checking.

This is apart from performance testing (performance in games), which is really another topic. I use cutechess-cli for that, driven by a bunch of shell scripts.

--Jon

Re: Basic automated testing

Posted: Mon Oct 01, 2018 3:53 am
by cdani
AndrewGrant wrote:
Sun Sep 30, 2018 10:51 pm
mar wrote:
Sun Sep 30, 2018 8:40 pm
Ratosh wrote:
Fri Sep 28, 2018 6:31 pm
I'd recommend OpenBench.
Interesting, what does this OpenBench do exactly? It seems like a wrapper on top of cutechess-cli with some client-server functionality.
If it allows me say to test on say two quads at the same time + gather the results then this would cut my testing time in half,
which sounds really interesting.
My instance is running here http://chess.grantnet.us/index/ , if you want to get an idea of the work flow.

You would be able to connect any of your machines, and the results will be collected together.

Time controls will be scaled per CPU, IE in the case that one machine is faster or something of the sort.

In case its not clear, I wrote OpenBench
I knew you have it, but I find is good time to congratulate you for your nice and interesting work! :-)

Re: Basic automated testing

Posted: Mon Oct 01, 2018 5:14 am
by odomobo
jdart wrote:
Mon Oct 01, 2018 3:21 am
I have a bunch of unit tests that include basic functionality. One of the components is perft (https://www.chessprogramming.org/Perft), which is quite useful for sanity checking.

This is apart from performance testing (performance in games), which is really another topic. I use cutechess-cli for that, driven by a bunch of shell scripts.

--Jon
What do your cutechess-cli scripts do? This might be what I am looking for (OpenBench looks too advanced for my current needs)

Re: Basic automated testing

Posted: Mon Oct 01, 2018 1:37 pm
by mar
AndrewGrant wrote:
Sun Sep 30, 2018 10:51 pm
My instance is running here http://chess.grantnet.us/index/ , if you want to get an idea of the work flow.

You would be able to connect any of your machines, and the results will be collected together.

Time controls will be scaled per CPU, IE in the case that one machine is faster or something of the sort.

In case its not clear, I wrote OpenBench
That looks really nice. How do you supply test binaries? It seems you use branches in git so I assume it just fetches the branches and rebuilds on client?

Re: Basic automated testing

Posted: Mon Oct 01, 2018 6:13 pm
by AndrewGrant
mar wrote:
Mon Oct 01, 2018 1:37 pm
AndrewGrant wrote:
Sun Sep 30, 2018 10:51 pm
My instance is running here http://chess.grantnet.us/index/ , if you want to get an idea of the work flow.

You would be able to connect any of your machines, and the results will be collected together.

Time controls will be scaled per CPU, IE in the case that one machine is faster or something of the sort.

In case its not clear, I wrote OpenBench
That looks really nice. How do you supply test binaries? It seems you use branches in git so I assume it just fetches the branches and rebuilds on client?
Correct. OpenBench assumes that gcc/g++ and make are on the system path. It will pull down a zip file from github -- no actual git comments are taking place.

Re: Basic automated testing

Posted: Mon Oct 01, 2018 6:22 pm
by AndrewGrant
cdani wrote:
Mon Oct 01, 2018 3:53 am
AndrewGrant wrote:
Sun Sep 30, 2018 10:51 pm
mar wrote:
Sun Sep 30, 2018 8:40 pm
Ratosh wrote:
Fri Sep 28, 2018 6:31 pm
I'd recommend OpenBench.
Interesting, what does this OpenBench do exactly? It seems like a wrapper on top of cutechess-cli with some client-server functionality.
If it allows me say to test on say two quads at the same time + gather the results then this would cut my testing time in half,
which sounds really interesting.
My instance is running here http://chess.grantnet.us/index/ , if you want to get an idea of the work flow.

You would be able to connect any of your machines, and the results will be collected together.

Time controls will be scaled per CPU, IE in the case that one machine is faster or something of the sort.

In case its not clear, I wrote OpenBench
I knew you have it, but I find is good time to congratulate you for your nice and interesting work! :-)
Thanks. The same can be said for Andscacs :)

Re: Basic automated testing

Posted: Tue Oct 02, 2018 10:27 pm
by jdart
What do your cutechess-cli scripts do? This might be what I am looking for (OpenBench looks too advanced for my current needs)
They are pretty simple. They distribute a compiled binary to a set of directories on a set of networked machines. Then they run a bunch of parallel matches on each machine. Then they gather all the completed games into a big game file and run BayesELO on it.

Unfortunately I have not bothered to make them very portable or reusable, so they have some hardcoded machine names, paths, etc.

--Jon