Test accuracy of chess engines

Discussion of chess software programming and technical issues.

Moderators: hgm, Rebel, chrisw

Chessnut1071
Posts: 313
Joined: Tue Aug 03, 2021 2:41 pm
Full name: Bill Beame

Test accuracy of chess engines

Post by Chessnut1071 »

I'm wondering how those of you who are developing your own chess engines are making sure they are error free. I initially thought it was fairly simple; however, after testing my engine's accuracy I constantly found errors. At first my engine could examine about 80,000 legal moves/sec-using an 8th gen i7 Intel chip, about 2.7GHz. I used 250 chess puzzles where the solution was known ranging from 3 - 9 move checkmates. I probably found around 1,000 errors over 11 months, bringing the speed of my engine down to a little under 20,000 legal moves/sec. It was a very humbling experience. I never imagined it would be that challenging.

So, my question is, what are you using to test your engines if its not chess puzzles, with known solution?
Ferdy
Posts: 4833
Joined: Sun Aug 10, 2008 3:15 pm
Location: Philippines

Re: Test accuracy of chess engines

Post by Ferdy »

Play engine vs engine matches. If it is stronger it is more accurate.
Or test your engine with positions having more solutions and different points like in MEA.
JVMerlino
Posts: 1357
Joined: Wed Mar 08, 2006 10:15 pm
Location: San Francisco, California

Re: Test accuracy of chess engines

Post by JVMerlino »

Chessnut1071 wrote: Sun Aug 15, 2021 6:37 am I'm wondering how those of you who are developing your own chess engines are making sure they are error free. I initially thought it was fairly simple; however, after testing my engine's accuracy I constantly found errors. At first my engine could examine about 80,000 legal moves/sec-using an 8th gen i7 Intel chip, about 2.7GHz. I used 250 chess puzzles where the solution was known ranging from 3 - 9 move checkmates. I probably found around 1,000 errors over 11 months, bringing the speed of my engine down to a little under 20,000 legal moves/sec. It was a very humbling experience. I never imagined it would be that challenging.

So, my question is, what are you using to test your engines if its not chess puzzles, with known solution?
First of all, are you certain that your move generation is 100% correct? Implementing perft and testing on a lot of tricky positions could find any number of errors. Try the ones on the computerchess wiki, and then these as well:
r3k2r/8/8/8/3pPp2/8/8/R3K1RR b KQkq e3 0 1 // perft(6) = 485,647,607
r3k2r/Pppp1ppp/1b3nbN/nP6/BBP1P3/q4N2/Pp1P2PP/R2Q1RK1 w kq - 0 1 // perft(6) = 706,045,033
8/7p/p5pb/4k3/P1pPn3/8/P5PP/1rB2RK1 b - d3 0 28 // perft(6) = 38,633,283
8/3K4/2p5/p2b2r1/5k2/8/8/1q6 b - - 1 67 // perft(7) = 493,407,574
rnbqkb1r/ppppp1pp/7n/4Pp2/8/8/PPPP1PPP/RNBQKBNR w KQkq f6 0 3 // perft(6) = 244,063,299
r3k2r/p1ppqpb1/bn2pnp1/3PN3/1p2P3/2N2Q1p/PPPBBPPP/R3K2R w KQkq - // Perft(5) = 193,690,690
8/p7/8/1P6/K1k3p1/6P1/7P/8 w - - // Perft(8) = 8,103,790
n1n5/PPPk4/8/8/8/8/4Kppp/5N1N b - - // Perft(6) = 71,179,139
r3k2r/p6p/8/B7/1pp1p3/3b4/P6P/R3K2R w KQkq - // Perft(6) = 77,054,993
8/2p5/3p4/KP5r/1R3p1k/8/4P1P1/8 w - - // Perft(7) = 178,633,661
8/5p2/8/2k3P1/p3K3/8/1P6/8 b - - // Perft(8) = 64,451,405
r3k2r/pb3p2/5npp/n2p4/1p1PPB2/6P1/P2N1PBP/R3K2R w KQkq - // Perft(5) = 29,179,893

Next, I'm quite concerned about your engine's speed. 20K moves/sec is extremely slow, as even most of the weaker engines can do at least 100K. Can you describe a bit about your engine's architecture? Language? Board representation? Basic search algorithm?
John Merlino - Myrddin chess engine