TalkChess.com

I have a question for everyone: what is your basic approach to automated testing? What tools do you use? Do you test engine strength only, or do you test for other things as well?

I want to get a workflow set up where if I make a change in the evening, I can click a button before I go to sleep and know by morning if I broke anything critical, and how much of an improvement the change made.

Problem with writing many test cases is that you have to maintain them. So if your interfaces change you have to rewrite many test cases as well.

So it is not for lazy developers with changing ideas every other day.

By the way if you write test cases at the end of development when interfaces are stable you are too late.

I usually write the least number of test cases. The less code the better. (*yawn*)

I'd recommend OpenBench.

Looks interesting. Can you describe its use a little more ?

I use tests embedded into the my engine. The tests can be found in the hedgehog source code. But it depends on what you are expecting to test out.

Ratosh wrote: ↑Fri Sep 28, 2018 8:31 pm I'd recommend OpenBench.

Interesting, what does this OpenBench do exactly? It seems like a wrapper on top of cutechess-cli with some client-server functionality.
If it allows me say to test on say two quads at the same time + gather the results then this would cut my testing time in half,
which sounds really interesting.

mar wrote: ↑Sun Sep 30, 2018 10:40 pm

Ratosh wrote: ↑Fri Sep 28, 2018 8:31 pm I'd recommend OpenBench.

Interesting, what does this OpenBench do exactly? It seems like a wrapper on top of cutechess-cli with some client-server functionality.
If it allows me say to test on say two quads at the same time + gather the results then this would cut my testing time in half,
which sounds really interesting.

My instance is running here http://chess.grantnet.us/index/ , if you want to get an idea of the work flow.

You would be able to connect any of your machines, and the results will be collected together.

Time controls will be scaled per CPU, IE in the case that one machine is faster or something of the sort.

In case its not clear, I wrote OpenBench

I have a bunch of unit tests that include basic functionality. One of the components is perft (https://www.chessprogramming.org/Perft), which is quite useful for sanity checking.

This is apart from performance testing (performance in games), which is really another topic. I use cutechess-cli for that, driven by a bunch of shell scripts.

--Jon

AndrewGrant wrote: ↑Mon Oct 01, 2018 12:51 am

mar wrote: ↑Sun Sep 30, 2018 10:40 pm

Ratosh wrote: ↑Fri Sep 28, 2018 8:31 pm I'd recommend OpenBench.

Interesting, what does this OpenBench do exactly? It seems like a wrapper on top of cutechess-cli with some client-server functionality.
If it allows me say to test on say two quads at the same time + gather the results then this would cut my testing time in half,
which sounds really interesting.

My instance is running here http://chess.grantnet.us/index/ , if you want to get an idea of the work flow.

You would be able to connect any of your machines, and the results will be collected together.

Time controls will be scaled per CPU, IE in the case that one machine is faster or something of the sort.

In case its not clear, I wrote OpenBench

I knew you have it, but I find is good time to congratulate you for your nice and interesting work!

jdart wrote: ↑Mon Oct 01, 2018 5:21 am I have a bunch of unit tests that include basic functionality. One of the components is perft (https://www.chessprogramming.org/Perft), which is quite useful for sanity checking.

This is apart from performance testing (performance in games), which is really another topic. I use cutechess-cli for that, driven by a bunch of shell scripts.

--Jon

What do your cutechess-cli scripts do? This might be what I am looking for (OpenBench looks too advanced for my current needs)