Basic automated testing

Discussion of chess software programming and technical issues.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
odomobo
Posts: 59
Joined: Thu Jul 05, 2018 11:09 pm
Location: Chicago, IL
Full name: Josh Odom

Basic automated testing

Post by odomobo » Fri Sep 28, 2018 4:07 pm

I have a question for everyone: what is your basic approach to automated testing? What tools do you use? Do you test engine strength only, or do you test for other things as well?

I want to get a workflow set up where if I make a change in the evening, I can click a button before I go to sleep and know by morning if I broke anything critical, and how much of an improvement the change made.

Henk
Posts: 5833
Joined: Mon May 27, 2013 8:31 am

Re: Basic automated testing

Post by Henk » Fri Sep 28, 2018 4:19 pm

Problem with writing many test cases is that you have to maintain them. So if your interfaces change you have to rewrite many test cases as well.

So it is not for lazy developers with changing ideas every other day.

By the way if you write test cases at the end of development when interfaces are stable you are too late.

I usually write the least number of test cases. The less code the better. (*yawn*)

Ratosh
Posts: 71
Joined: Mon Apr 16, 2018 4:56 pm

Re: Basic automated testing

Post by Ratosh » Fri Sep 28, 2018 6:31 pm

I'd recommend OpenBench.

User avatar
xr_a_y
Posts: 785
Joined: Sat Nov 25, 2017 1:28 pm
Location: France

Re: Basic automated testing

Post by xr_a_y » Fri Sep 28, 2018 7:31 pm

Looks interesting. Can you describe its use a little more ?

User avatar
Kotlov
Posts: 209
Joined: Fri Jul 10, 2015 7:23 pm
Location: Russia

Re: Basic automated testing

Post by Kotlov » Sun Sep 30, 2018 5:53 pm

I use tests embedded into the my engine. The tests can be found in the hedgehog source code. But it depends on what you are expecting to test out.
Eugene Kotlov
Hedgehog 2.0 64-bit coming soon...

mar
Posts: 2010
Joined: Fri Nov 26, 2010 1:00 pm
Location: Czech Republic
Full name: Martin Sedlak

Re: Basic automated testing

Post by mar » Sun Sep 30, 2018 8:40 pm

Ratosh wrote:
Fri Sep 28, 2018 6:31 pm
I'd recommend OpenBench.
Interesting, what does this OpenBench do exactly? It seems like a wrapper on top of cutechess-cli with some client-server functionality.
If it allows me say to test on say two quads at the same time + gather the results then this would cut my testing time in half,
which sounds really interesting.
Martin Sedlak

AndrewGrant
Posts: 494
Joined: Tue Apr 19, 2016 4:08 am
Location: U.S.A
Full name: Andrew Grant
Contact:

Re: Basic automated testing

Post by AndrewGrant » Sun Sep 30, 2018 10:51 pm

mar wrote:
Sun Sep 30, 2018 8:40 pm
Ratosh wrote:
Fri Sep 28, 2018 6:31 pm
I'd recommend OpenBench.
Interesting, what does this OpenBench do exactly? It seems like a wrapper on top of cutechess-cli with some client-server functionality.
If it allows me say to test on say two quads at the same time + gather the results then this would cut my testing time in half,
which sounds really interesting.
My instance is running here http://chess.grantnet.us/index/ , if you want to get an idea of the work flow.

You would be able to connect any of your machines, and the results will be collected together.

Time controls will be scaled per CPU, IE in the case that one machine is faster or something of the sort.

In case its not clear, I wrote OpenBench

jdart
Posts: 3837
Joined: Fri Mar 10, 2006 4:23 am
Location: http://www.arasanchess.org

Re: Basic automated testing

Post by jdart » Mon Oct 01, 2018 3:21 am

I have a bunch of unit tests that include basic functionality. One of the components is perft (https://www.chessprogramming.org/Perft), which is quite useful for sanity checking.

This is apart from performance testing (performance in games), which is really another topic. I use cutechess-cli for that, driven by a bunch of shell scripts.

--Jon

User avatar
cdani
Posts: 2104
Joined: Sat Jan 18, 2014 9:24 am
Location: Andorra
Contact:

Re: Basic automated testing

Post by cdani » Mon Oct 01, 2018 3:53 am

AndrewGrant wrote:
Sun Sep 30, 2018 10:51 pm
mar wrote:
Sun Sep 30, 2018 8:40 pm
Ratosh wrote:
Fri Sep 28, 2018 6:31 pm
I'd recommend OpenBench.
Interesting, what does this OpenBench do exactly? It seems like a wrapper on top of cutechess-cli with some client-server functionality.
If it allows me say to test on say two quads at the same time + gather the results then this would cut my testing time in half,
which sounds really interesting.
My instance is running here http://chess.grantnet.us/index/ , if you want to get an idea of the work flow.

You would be able to connect any of your machines, and the results will be collected together.

Time controls will be scaled per CPU, IE in the case that one machine is faster or something of the sort.

In case its not clear, I wrote OpenBench
I knew you have it, but I find is good time to congratulate you for your nice and interesting work! :-)

odomobo
Posts: 59
Joined: Thu Jul 05, 2018 11:09 pm
Location: Chicago, IL
Full name: Josh Odom

Re: Basic automated testing

Post by odomobo » Mon Oct 01, 2018 5:14 am

jdart wrote:
Mon Oct 01, 2018 3:21 am
I have a bunch of unit tests that include basic functionality. One of the components is perft (https://www.chessprogramming.org/Perft), which is quite useful for sanity checking.

This is apart from performance testing (performance in games), which is really another topic. I use cutechess-cli for that, driven by a bunch of shell scripts.

--Jon
What do your cutechess-cli scripts do? This might be what I am looking for (OpenBench looks too advanced for my current needs)

Post Reply