Debugging for lazy chess programmers

Discussion of chess software programming and technical issues.

Moderators: hgm, Harvey Williamson, bob

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
Henk
Posts: 5294
Joined: Mon May 27, 2013 8:31 am

Debugging for lazy chess programmers

Post by Henk » Fri Sep 06, 2013 3:36 pm

Again I have been busy all day with debugging. This time there was a bug related to updating bit boards when capturing pieces of the same type (Rook x Rook, Pawn x Pawn etc)

So I expect there will be more of these tough bugs in my software. The problem with my latest bug is that is wasn't visible at the user level. It played normal chess. Only by accidence when I tried to optimize my q-search I encountered them.

Removing the bug did not make my program play better. So perhaps retuning (again and again) will be necessary.

To reduce bugs one may use:

- Perft
- Code inspection
- Test cases

I don't like writing test cases. What other methods are there for finding hidden bugs or the bugs you cannot find and you'll always overlook ?

Gerd Isenberg
Posts: 2110
Joined: Wed Mar 08, 2006 7:47 pm
Location: Hattingen, Germany

Re: Debugging for lazy chess programmers

Post by Gerd Isenberg » Fri Sep 06, 2013 4:08 pm

Henk wrote:Again I have been busy all day with debugging. This time there was a bug related to updating bit boards when capturing pieces of the same type (Rook x Rook, Pawn x Pawn etc)

So I expect there will be more of these tough bugs in my software. The problem with my latest bug is that is wasn't visible at the user level. It played normal chess. Only by accidence when I tried to optimize my q-search I encountered them.

Removing the bug did not make my program play better. So perhaps retuning (again and again) will be necessary.

To reduce bugs one may use:

- Perft
- Code inspection
- Test cases

I don't like writing test cases. What other methods are there for finding hidden bugs or the bugs you cannot find and you'll always overlook ?
Checking pre- and post-conditions via assert in debug mode. Comparing incremental updated datastructures with initialization from scratch, i.e. hashkeys and board representation. Testing color-flipped positions. Logging part of search trees with triggers, or conditional breakpoints if hashkey (position) matches some given value.

User avatar
pocopito
Posts: 230
Joined: Tue Jul 12, 2011 11:31 am
Contact:

Re: Debugging for lazy chess programmers

Post by pocopito » Fri Sep 06, 2013 4:39 pm

Maybe it was a coincidence, but a couple of times I was aware of the existence of bugs in my code because different compile options gave place to a different number of nodes. For example a plain "gcc -o my_engine *.c" vs "gcc -O3 -o myengine *.c".
At least this situation arose twice; in one occasion there was a bug in my eval and in another occasion there was a bug in the search code.

Best

E Diaz
Two first meanings of the dutch word "leren":
1. leren [vc] (learn, larn, acquire) acquire or gain knowledge or skills.
2. leren [v] (teach, learn, instruct) impart skills or knowledge to.

tpetzke
Posts: 685
Joined: Thu Mar 03, 2011 3:57 pm
Location: Germany
Contact:

Re: Debugging for lazy chess programmers

Post by tpetzke » Fri Sep 06, 2013 4:58 pm

I hit the same issue several times, different node counts between Debug and Release, between x32 and x64 and Intel vs MSVC.

Was painful to hunt it down but at the end there was always a bug :-)

Thomas...
Thomas...

=======
http://macechess.blogspot.com - iCE Chess Engine

Ferdy
Posts: 3620
Joined: Sun Aug 10, 2008 1:15 pm
Location: Philippines

Re: Debugging for lazy chess programmers

Post by Ferdy » Fri Sep 06, 2013 6:11 pm

In the eval for example, you coded a white pawn in e-file at 4th rank as passer, debug it by inserting a print_board() or print_fen(), run the program and see if it is really a passer :) . Added to that you can even feel how much its worth in the overall evaluation. This is very effective when designing a kingsafety evaluation - strong players have the advantage of this method.

BeyondCritics
Posts: 335
Joined: Sat May 05, 2012 12:48 pm
Location: Bergheim

Re: Debugging for lazy chess programmers

Post by BeyondCritics » Fri Sep 06, 2013 6:33 pm

Hi Henrik,

i have precollected the following urls about testing tools. They are looking promising but admittedly i have tried none of them so far:
http://ccadar.github.io/klee/
http://research.microsoft.com/en-us/projects/pex/
http://code.google.com/p/crest/

Regression testing is relative cheap.

You can try tools for static analysis of code. From my experience as professional tester i know, that PC-Lint is still widely regarded as having the most reasonable cost-benefit ratio, but it is not a bargain and using it can be work, since you may get false positives.

In my experience, books about software engineering can get rather dry, so you may get the wrong impression from them. I like this one: http://www.amazon.com/Software-Testing- ... 0849374758
A well thought out test can be intellectual satisfying. And you get the gut feeling that your software is correct.

Best Regards
Oliver

User avatar
Codesquid
Posts: 138
Joined: Tue Aug 23, 2011 8:25 pm
Location: Germany
Contact:

Re: Debugging for lazy chess programmers

Post by Codesquid » Fri Sep 06, 2013 6:50 pm

Good way to spot bugs:

Make your program 100% deterministic (e.g. replace std::sort with std::stable sort, add a feature to specify which random seed to use and so on), then search a position to some depth (e.g. 20 ply) using a single core and write down the total number of nodes visited.
Next, compile it using an entirely different compiler from a different vendor (e.g. GCC if you usually use Visual Studio) and run it with the same parameters. If node count is not identical, you have a very subtle bug somewhere.
nanos gigantium humeris insidentes

jdart
Posts: 3618
Joined: Fri Mar 10, 2006 4:23 am
Location: http://www.arasanchess.org

Re: Debugging for lazy chess programmers

Post by jdart » Sat Sep 07, 2013 2:06 am

assert is very useful. I do a lot of things via assert that are not feasible in a normal, non-debug build, such as verifying that the side to move is really in check when computing evasion moves, verifying that moves from the hashtable are legal, verifying incremental update of the hash function equals what you get from a from-scratch computation, checking various array bound limits, etc. All these are quite expensive and slow the search down, but in debug mode they will catch a lot of errors, before you get a memory overflow or something else that will crash the program in a state that is hard to debug.

You can argue this is the lazy man's way to do checking and it shouldn't really be necessary if you have debugged code properly as it was going into the codebase. But it has saved my butt quite a few times.

--Jon

sedicla
Posts: 143
Joined: Fri Jan 07, 2011 11:51 pm
Location: USA

Re: Debugging for lazy chess programmers

Post by sedicla » Sat Sep 07, 2013 1:31 pm

I have the following test commands in my engine:

- perft: regular move count from current position.
- perftx: detailed perf with additional counts of captures, castle, promotions, etc, for a couple of starting positions. Based on the wiki programming page about perft.
- perfty: another perft version based on different positions and depth counts.
- evtest: read an epd file and flip the board right to left, up and down to make sure the evaluation is symmetric. you can catch some bugs here too. You will need an epd file for that.
- bench: count the nodes for 12 positions and print the total. It is good for testing non functional changes. This works if you have a deterministic program as mentioned in this thread.

Of course lot of asserts in the code.

Hope this can help. Good luck with the bug hunting... :)

My engine is in https://sites.google.com/site/tucanochess/

User avatar
Steve Maughan
Posts: 1042
Joined: Wed Mar 08, 2006 7:28 pm
Location: Florida, USA
Contact:

Re: Debugging for lazy chess programmers

Post by Steve Maughan » Wed Sep 11, 2013 12:05 am

Hi Thomas,

I took your advice and compared release vs. debug versions. There was indeed a bug. You can see my blog post about it here:

http://www.chessprogramming.net/compute ... g-hunting/

Cool technique!

Thanks,

Steve
http://www.chessprogramming.net - Maverick Chess Engine

Post Reply