Debugging for lazy chess programmers

[Henk]

Again I have been busy all day with debugging. This time there was a bug related to updating bit boards when capturing pieces of the same type (Rook x Rook, Pawn x Pawn etc)

So I expect there will be more of these tough bugs in my software. The problem with my latest bug is that is wasn't visible at the user level. It played normal chess. Only by accidence when I tried to optimize my q-search I encountered them.

Removing the bug did not make my program play better. So perhaps retuning (again and again) will be necessary.

To reduce bugs one may use:

- Perft
- Code inspection
- Test cases

I don't like writing test cases. What other methods are there for finding hidden bugs or the bugs you cannot find and you'll always overlook ?

[Gerd Isenberg]

Again I have been busy all day with debugging. This time there was a bug related to updating bit boards when capturing pieces of the same type (Rook x Rook, Pawn x Pawn etc)

So I expect there will be more of these tough bugs in my software. The problem with my latest bug is that is wasn't visible at the user level. It played normal chess. Only by accidence when I tried to optimize my q-search I encountered them.

Removing the bug did not make my program play better. So perhaps retuning (again and again) will be necessary.

To reduce bugs one may use:

- Perft
- Code inspection
- Test cases

I don't like writing test cases. What other methods are there for finding hidden bugs or the bugs you cannot find and you'll always overlook ?

Checking pre- and post-conditions via assert in debug mode. Comparing incremental updated datastructures with initialization from scratch, i.e. hashkeys and board representation. Testing color-flipped positions. Logging part of search trees with triggers, or conditional breakpoints if hashkey (position) matches some given value.

[pocopito]

Maybe it was a coincidence, but a couple of times I was aware of the existence of bugs in my code because different compile options gave place to a different number of nodes. For example a plain "gcc -o my_engine *.c" vs "gcc -O3 -o myengine *.c".
At least this situation arose twice; in one occasion there was a bug in my eval and in another occasion there was a bug in the search code.

Best

E Diaz

[tpetzke]

I hit the same issue several times, different node counts between Debug and Release, between x32 and x64 and Intel vs MSVC.

Was painful to hunt it down but at the end there was always a bug

Thomas...

[Ferdy]

In the eval for example, you coded a white pawn in e-file at 4th rank as passer, debug it by inserting a print_board() or print_fen(), run the program and see if it is really a passer . Added to that you can even feel how much its worth in the overall evaluation. This is very effective when designing a kingsafety evaluation - strong players have the advantage of this method.

[BeyondCritics]

Hi Henrik,

i have precollected the following urls about testing tools. They are looking promising but admittedly i have tried none of them so far:
http://ccadar.github.io/klee/
http://research.microsoft.com/en-us/projects/pex/
http://code.google.com/p/crest/

Regression testing is relative cheap.

You can try tools for static analysis of code. From my experience as professional tester i know, that PC-Lint is still widely regarded as having the most reasonable cost-benefit ratio, but it is not a bargain and using it can be work, since you may get false positives.

In my experience, books about software engineering can get rather dry, so you may get the wrong impression from them. I like this one: http://www.amazon.com/Software-Testing- ... 0849374758
A well thought out test can be intellectual satisfying. And you get the gut feeling that your software is correct.

Best Regards
Oliver

[Codesquid]

Good way to spot bugs:

Make your program 100% deterministic (e.g. replace std::sort with std::stable sort, add a feature to specify which random seed to use and so on), then search a position to some depth (e.g. 20 ply) using a single core and write down the total number of nodes visited.
Next, compile it using an entirely different compiler from a different vendor (e.g. GCC if you usually use Visual Studio) and run it with the same parameters. If node count is not identical, you have a very subtle bug somewhere.

[jdart]

assert is very useful. I do a lot of things via assert that are not feasible in a normal, non-debug build, such as verifying that the side to move is really in check when computing evasion moves, verifying that moves from the hashtable are legal, verifying incremental update of the hash function equals what you get from a from-scratch computation, checking various array bound limits, etc. All these are quite expensive and slow the search down, but in debug mode they will catch a lot of errors, before you get a memory overflow or something else that will crash the program in a state that is hard to debug.

You can argue this is the lazy man's way to do checking and it shouldn't really be necessary if you have debugged code properly as it was going into the codebase. But it has saved my butt quite a few times.

--Jon

[sedicla]

I have the following test commands in my engine:

- perft: regular move count from current position.
- perftx: detailed perf with additional counts of captures, castle, promotions, etc, for a couple of starting positions. Based on the wiki programming page about perft.
- perfty: another perft version based on different positions and depth counts.
- evtest: read an epd file and flip the board right to left, up and down to make sure the evaluation is symmetric. you can catch some bugs here too. You will need an epd file for that.
- bench: count the nodes for 12 positions and print the total. It is good for testing non functional changes. This works if you have a deterministic program as mentioned in this thread.

Of course lot of asserts in the code.

Hope this can help. Good luck with the bug hunting...

My engine is in https://sites.google.com/site/tucanochess/

[Steve Maughan]

Hi Thomas,

I took your advice and compared release vs. debug versions. There was indeed a bug. You can see my blog post about it here:

http://www.chessprogramming.net/compute ... g-hunting/

Cool technique!

Thanks,

Steve