Strategies to unit testing the search

[osvitashev]

I am interested in being able to write unit tests for my engine, which is in pretty early stage at this point.

I am trying to follow the idea of having a lot of simple tests for all of the individual components of the engine. What's known as Testing Pyramid in the software industry.
At this point i have:
1. Perft-based tests to verify move generation.
2. Position evaluation tests which run evaluation function on a given position.
3. Mate-in-X-moves tests which verify my negamax/alphabeta search. (Arguably, this is more of an integration-level test)

Next on my list is to add features which control the way game tree is explored. (e.g. Killer Move and Quiescence Search)
So here is the main question:
How do i test that these algorithms independently?

From i see around, a common approach for testing is to set up a test position and verify that the correct move is returned form the search.
This works but I feel such an approach lacks granularity, and does not make it easy to locate the cause of potential failures.

Ultimately, i am trying to avoid a situation where changes to evaluation function lead to failing tests, but i need to figure out whether it merely exposed a bug somewhere else.

Did anyone come up with an approach to test search algorithm optimizations granularly?
Or, perhaps, the answer is that the recursive nature of chess search does not make it easy to have proper unit tests and the best strategy is to just have the engine play against itself to check for regressions?

[Bo Persson]

Next on my list is to add features which control the way game tree is explored. (e.g. Killer Move and Quiescence Search)
So here is the main question:
How do i test that these algorithms independently?

You probably don't.

Well, you can of course test that the Killer Move handler updates and retrieves the stored moves correctly. But not how it affects the search. Because that is no longer a unit test, but a system integration test.

Now you might no longer set up simple test cases to see if a different weight in killer moves is good or bad, when the hash move cuts the tree before the killer move is even used.

[Fabio Gobbato]

You can take your search function and remove all the pruning and make it like a perft to check if hash move, killer moves, countermove and move ordering doesn't play 2 identical moves or skip some moves.
Could be useful also a unit test for SEE.
To test other parts fill the search with asserts you can find some weird errors.

[jswaff]

I am interested in being able to write unit tests for my engine, which is in pretty early stage at this point.

I am trying to follow the idea of having a lot of simple tests for all of the individual components of the engine. What's known as Testing Pyramid in the software industry.
At this point i have:
1. Perft-based tests to verify move generation.
2. Position evaluation tests which run evaluation function on a given position.
3. Mate-in-X-moves tests which verify my negamax/alphabeta search. (Arguably, this is more of an integration-level test)

Next on my list is to add features which control the way game tree is explored. (e.g. Killer Move and Quiescence Search)
So here is the main question:
How do i test that these algorithms independently?

From i see around, a common approach for testing is to set up a test position and verify that the correct move is returned form the search.
This works but I feel such an approach lacks granularity, and does not make it easy to locate the cause of potential failures.

Ultimately, i am trying to avoid a situation where changes to evaluation function lead to failing tests, but i need to figure out whether it merely exposed a bug somewhere else.

Did anyone come up with an approach to test search algorithm optimizations granularly?
Or, perhaps, the answer is that the recursive nature of chess search does not make it easy to have proper unit tests and the best strategy is to just have the engine play against itself to check for regressions?

I did some basic unit testing using a mocking framework in my Java program chess4j to test some really basic functionality, but past that you probably want to use another testing strategy. Test suites will tell you if you've broken anything too badly. Past that I think you just play games.

https://github.com/jswaff/chess4j/blob/ ... hTest.java

[jdart]

Besides perft, I have some simple unit tests that check for expected moves from some fairly simple search positions. They won't catch all bugs but are more of a sanity check. See method testSearch in https://github.com/jdart1/arasan-chess/ ... c/unit.cpp.