Page 1 of 2

Basic automated testing

Posted: Fri Sep 28, 2018 4:07 pm
by odomobo
I have a question for everyone: what is your basic approach to automated testing? What tools do you use? Do you test engine strength only, or do you test for other things as well?

I want to get a workflow set up where if I make a change in the evening, I can click a button before I go to sleep and know by morning if I broke anything critical, and how much of an improvement the change made.

Re: Basic automated testing

Posted: Fri Sep 28, 2018 4:19 pm
by Henk
Problem with writing many test cases is that you have to maintain them. So if your interfaces change you have to rewrite many test cases as well.

So it is not for lazy developers with changing ideas every other day.

By the way if you write test cases at the end of development when interfaces are stable you are too late.

I usually write the least number of test cases. The less code the better. (*yawn*)

Re: Basic automated testing

Posted: Fri Sep 28, 2018 6:31 pm
by Ratosh
I'd recommend OpenBench.

Re: Basic automated testing

Posted: Fri Sep 28, 2018 7:31 pm
by xr_a_y
Looks interesting. Can you describe its use a little more ?

Re: Basic automated testing

Posted: Sun Sep 30, 2018 5:53 pm
by Kotlov
I use tests embedded into the my engine. The tests can be found in the hedgehog source code. But it depends on what you are expecting to test out.

Re: Basic automated testing

Posted: Sun Sep 30, 2018 8:40 pm
by mar
Ratosh wrote:
Fri Sep 28, 2018 6:31 pm
I'd recommend OpenBench.
Interesting, what does this OpenBench do exactly? It seems like a wrapper on top of cutechess-cli with some client-server functionality.
If it allows me say to test on say two quads at the same time + gather the results then this would cut my testing time in half,
which sounds really interesting.

Re: Basic automated testing

Posted: Sun Sep 30, 2018 10:51 pm
by AndrewGrant
mar wrote:
Sun Sep 30, 2018 8:40 pm
Ratosh wrote:
Fri Sep 28, 2018 6:31 pm
I'd recommend OpenBench.
Interesting, what does this OpenBench do exactly? It seems like a wrapper on top of cutechess-cli with some client-server functionality.
If it allows me say to test on say two quads at the same time + gather the results then this would cut my testing time in half,
which sounds really interesting.
My instance is running here http://chess.grantnet.us/index/ , if you want to get an idea of the work flow.

You would be able to connect any of your machines, and the results will be collected together.

Time controls will be scaled per CPU, IE in the case that one machine is faster or something of the sort.

In case its not clear, I wrote OpenBench

Re: Basic automated testing

Posted: Mon Oct 01, 2018 3:21 am
by jdart
I have a bunch of unit tests that include basic functionality. One of the components is perft (https://www.chessprogramming.org/Perft), which is quite useful for sanity checking.

This is apart from performance testing (performance in games), which is really another topic. I use cutechess-cli for that, driven by a bunch of shell scripts.

--Jon

Re: Basic automated testing

Posted: Mon Oct 01, 2018 3:53 am
by cdani
AndrewGrant wrote:
Sun Sep 30, 2018 10:51 pm
mar wrote:
Sun Sep 30, 2018 8:40 pm
Ratosh wrote:
Fri Sep 28, 2018 6:31 pm
I'd recommend OpenBench.
Interesting, what does this OpenBench do exactly? It seems like a wrapper on top of cutechess-cli with some client-server functionality.
If it allows me say to test on say two quads at the same time + gather the results then this would cut my testing time in half,
which sounds really interesting.
My instance is running here http://chess.grantnet.us/index/ , if you want to get an idea of the work flow.

You would be able to connect any of your machines, and the results will be collected together.

Time controls will be scaled per CPU, IE in the case that one machine is faster or something of the sort.

In case its not clear, I wrote OpenBench
I knew you have it, but I find is good time to congratulate you for your nice and interesting work! :-)

Re: Basic automated testing

Posted: Mon Oct 01, 2018 5:14 am
by odomobo
jdart wrote:
Mon Oct 01, 2018 3:21 am
I have a bunch of unit tests that include basic functionality. One of the components is perft (https://www.chessprogramming.org/Perft), which is quite useful for sanity checking.

This is apart from performance testing (performance in games), which is really another topic. I use cutechess-cli for that, driven by a bunch of shell scripts.

--Jon
What do your cutechess-cli scripts do? This might be what I am looking for (OpenBench looks too advanced for my current needs)