Debugging for lazy chess programmers

Discussion of chess software programming and technical issues.

Moderators: hgm, Rebel, chrisw

Henk
Posts: 7226
Joined: Mon May 27, 2013 10:31 am

Debugging for lazy chess programmers

Post by Henk »

Again I have been busy all day with debugging. This time there was a bug related to updating bit boards when capturing pieces of the same type (Rook x Rook, Pawn x Pawn etc)

So I expect there will be more of these tough bugs in my software. The problem with my latest bug is that is wasn't visible at the user level. It played normal chess. Only by accidence when I tried to optimize my q-search I encountered them.

Removing the bug did not make my program play better. So perhaps retuning (again and again) will be necessary.

To reduce bugs one may use:

- Perft
- Code inspection
- Test cases

I don't like writing test cases. What other methods are there for finding hidden bugs or the bugs you cannot find and you'll always overlook ?
Gerd Isenberg
Posts: 2251
Joined: Wed Mar 08, 2006 8:47 pm
Location: Hattingen, Germany

Re: Debugging for lazy chess programmers

Post by Gerd Isenberg »

Henk wrote:Again I have been busy all day with debugging. This time there was a bug related to updating bit boards when capturing pieces of the same type (Rook x Rook, Pawn x Pawn etc)

So I expect there will be more of these tough bugs in my software. The problem with my latest bug is that is wasn't visible at the user level. It played normal chess. Only by accidence when I tried to optimize my q-search I encountered them.

Removing the bug did not make my program play better. So perhaps retuning (again and again) will be necessary.

To reduce bugs one may use:

- Perft
- Code inspection
- Test cases

I don't like writing test cases. What other methods are there for finding hidden bugs or the bugs you cannot find and you'll always overlook ?
Checking pre- and post-conditions via assert in debug mode. Comparing incremental updated datastructures with initialization from scratch, i.e. hashkeys and board representation. Testing color-flipped positions. Logging part of search trees with triggers, or conditional breakpoints if hashkey (position) matches some given value.
User avatar
pocopito
Posts: 238
Joined: Tue Jul 12, 2011 1:31 pm

Re: Debugging for lazy chess programmers

Post by pocopito »

Maybe it was a coincidence, but a couple of times I was aware of the existence of bugs in my code because different compile options gave place to a different number of nodes. For example a plain "gcc -o my_engine *.c" vs "gcc -O3 -o myengine *.c".
At least this situation arose twice; in one occasion there was a bug in my eval and in another occasion there was a bug in the search code.

Best

E Diaz
Two first meanings of the dutch word "leren":
1. leren [vc] (learn, larn, acquire) acquire or gain knowledge or skills.
2. leren [v] (teach, learn, instruct) impart skills or knowledge to.
tpetzke
Posts: 686
Joined: Thu Mar 03, 2011 4:57 pm
Location: Germany

Re: Debugging for lazy chess programmers

Post by tpetzke »

I hit the same issue several times, different node counts between Debug and Release, between x32 and x64 and Intel vs MSVC.

Was painful to hunt it down but at the end there was always a bug :-)

Thomas...
Thomas...

=======
http://macechess.blogspot.com - iCE Chess Engine
Ferdy
Posts: 4840
Joined: Sun Aug 10, 2008 3:15 pm
Location: Philippines

Re: Debugging for lazy chess programmers

Post by Ferdy »

In the eval for example, you coded a white pawn in e-file at 4th rank as passer, debug it by inserting a print_board() or print_fen(), run the program and see if it is really a passer :) . Added to that you can even feel how much its worth in the overall evaluation. This is very effective when designing a kingsafety evaluation - strong players have the advantage of this method.
BeyondCritics
Posts: 406
Joined: Sat May 05, 2012 2:48 pm
Full name: Oliver Roese

Re: Debugging for lazy chess programmers

Post by BeyondCritics »

Hi Henrik,

i have precollected the following urls about testing tools. They are looking promising but admittedly i have tried none of them so far:
http://ccadar.github.io/klee/
http://research.microsoft.com/en-us/projects/pex/
http://code.google.com/p/crest/

Regression testing is relative cheap.

You can try tools for static analysis of code. From my experience as professional tester i know, that PC-Lint is still widely regarded as having the most reasonable cost-benefit ratio, but it is not a bargain and using it can be work, since you may get false positives.

In my experience, books about software engineering can get rather dry, so you may get the wrong impression from them. I like this one: http://www.amazon.com/Software-Testing- ... 0849374758
A well thought out test can be intellectual satisfying. And you get the gut feeling that your software is correct.

Best Regards
Oliver
User avatar
Codesquid
Posts: 138
Joined: Tue Aug 23, 2011 10:25 pm
Location: Germany

Re: Debugging for lazy chess programmers

Post by Codesquid »

Good way to spot bugs:

Make your program 100% deterministic (e.g. replace std::sort with std::stable sort, add a feature to specify which random seed to use and so on), then search a position to some depth (e.g. 20 ply) using a single core and write down the total number of nodes visited.
Next, compile it using an entirely different compiler from a different vendor (e.g. GCC if you usually use Visual Studio) and run it with the same parameters. If node count is not identical, you have a very subtle bug somewhere.
nanos gigantium humeris insidentes
jdart
Posts: 4368
Joined: Fri Mar 10, 2006 5:23 am
Location: http://www.arasanchess.org

Re: Debugging for lazy chess programmers

Post by jdart »

assert is very useful. I do a lot of things via assert that are not feasible in a normal, non-debug build, such as verifying that the side to move is really in check when computing evasion moves, verifying that moves from the hashtable are legal, verifying incremental update of the hash function equals what you get from a from-scratch computation, checking various array bound limits, etc. All these are quite expensive and slow the search down, but in debug mode they will catch a lot of errors, before you get a memory overflow or something else that will crash the program in a state that is hard to debug.

You can argue this is the lazy man's way to do checking and it shouldn't really be necessary if you have debugged code properly as it was going into the codebase. But it has saved my butt quite a few times.

--Jon
sedicla
Posts: 178
Joined: Sat Jan 08, 2011 12:51 am
Location: USA
Full name: Alcides Schulz

Re: Debugging for lazy chess programmers

Post by sedicla »

I have the following test commands in my engine:

- perft: regular move count from current position.
- perftx: detailed perf with additional counts of captures, castle, promotions, etc, for a couple of starting positions. Based on the wiki programming page about perft.
- perfty: another perft version based on different positions and depth counts.
- evtest: read an epd file and flip the board right to left, up and down to make sure the evaluation is symmetric. you can catch some bugs here too. You will need an epd file for that.
- bench: count the nodes for 12 positions and print the total. It is good for testing non functional changes. This works if you have a deterministic program as mentioned in this thread.

Of course lot of asserts in the code.

Hope this can help. Good luck with the bug hunting... :)

My engine is in https://sites.google.com/site/tucanochess/
User avatar
Steve Maughan
Posts: 1231
Joined: Wed Mar 08, 2006 8:28 pm
Location: Florida, USA

Re: Debugging for lazy chess programmers

Post by Steve Maughan »

Hi Thomas,

I took your advice and compared release vs. debug versions. There was indeed a bug. You can see my blog post about it here:

http://www.chessprogramming.net/compute ... g-hunting/

Cool technique!

Thanks,

Steve
http://www.chessprogramming.net - Maverick Chess Engine