AI performs poorly despite decent depth.

[AngularMomentum]

I implemented a chess engine and had it play a game against itself. It searched a maximum of 500 million nodes(takes about a minute) and searched to a depth of 8-10 in the middle game(7 was the lowest) not counting the quiescence search. I didn’t use any pruning that had a chance to change the result. My evaluation function is a piece square table with a 10cp side to move bonus. Despite the decent depth, the AI only had an accuracy of 75% according to chess.com. Is this normal, or is it likely that my program has a bug?

[Mike Sherwin]

I implemented a chess engine and had it play a game against itself. It searched a maximum of 500 million nodes(takes about a minute) and searched to a depth of 8-10 in the middle game(7 was the lowest) not counting the quiescence search. I didn’t use any pruning that had a chance to change the result. My evaluation function is a piece square table with a 10cp side to move bonus. Despite the decent depth, the AI only had an accuracy of 75% according to chess.com. Is this normal, or is it likely that my program has a bug?

Of course there can be bugs. We all wrestle with bugs. But 75% accuracy for such a minimal engine sounds quite good. And 8-10 depth is not a decent depth. Once the engine gets some decent move ordering then it will search to a decent depth.

[AngularMomentum]

The depth of 8-10 was in the most active part of the game(move 10-25) where accuracy wasn't even 50%. What depth should I be expecting with half a billion nodes without any pruning but good move ordering? I implemented principal variation search.

[AngularMomentum]

I implemented a chess engine and had it play a game against itself. It searched a maximum of 500 million nodes(takes about a minute) and searched to a depth of 8-10 in the middle game(7 was the lowest) not counting the quiescence search. I didn’t use any pruning that had a chance to change the result. My evaluation function is a piece square table with a 10cp side to move bonus. Despite the decent depth, the AI only had an accuracy of 75% according to chess.com. Is this normal, or is it likely that my program has a bug?

Of course there can be bugs. We all wrestle with bugs. But 75% accuracy for such a minimal engine sounds quite good. And 8-10 depth is not a decent depth. Once the engine gets some decent move ordering then it will search to a decent depth.

The depth of 8-10 was in the most active part of the game(move 10-25) where accuracy wasn't even 50%. What depth should I be expecting with half a billion nodes without any pruning but good move ordering? I implemented principal variation search.

[mvanthoor]

Is this normal, or is it likely that my program has a bug?

You have not provided any source code so nobody can see what you're doing. You also didn't provide a binary and a list of implemented features, so nobody can test your binary against other engines with the same features. Therefore nobody can give you a meaningful answer.

I'd be happy to take a look at the source code or run a test against my own engine so I can give a somewhat decent answer, but "I implemented a chess engine and it doesn't perform according to my epectations" is not enough information to go on.

[Mike Sherwin]

The depth of 8-10 was in the most active part of the game(move 10-25) where accuracy wasn't even 50%. What depth should I be expecting with half a billion nodes without any pruning but good move ordering? I implemented principal variation search.

My engine Bricabrac is a minimal engine right now. I disabled Null Move. It does not have decent move ordering yet. It does not have any other reductions at the moment. It has principle variation search (sort of) and a shallow search to order moves if depth > 3.

Begin Depth 12
12 37 5687 476792712 e2e4 e7e5 g1f3 g8f6 f3e5 f6e4 d1e2 d7d5 d2d3 f8b4 c2c3 b4d6

So in a little less than half a billion moves it reaches depth 12.

So with decent move ordering 14 to 16 i'd say.

[AngularMomentum]

Is this normal, or is it likely that my program has a bug?

You have not provided any source code so nobody can see what you're doing. You also didn't provide a binary and a list of implemented features, so nobody can test your binary against other engines with the same features. Therefore nobody can give you a meaningful answer.

I'd be happy to take a look at the source code or run a test against my own engine so I can give a somewhat decent answer, but "I implemented a chess engine and it doesn't perform according to my epectations" is not enough information to go on.

I obviously wasn’t asking anyone else to look at my code and find bugs for me. I was simply asking if the performance is normal for the features I implemented.
Move generation: 100-150 million nps in Perft, 15-20 million in actual search.
Search: PVS, quiescence search.
Move Ordering: MVV-LVA when the captured piece is more valuable than the moving piece, SEE for all other moves.
Evaluation: piece square table with 10cp side to move bonus.

[smatovic]

There is a known search:eval co-dependency, if you search deep but your eval is lame it won't work out, and, generally this depends on engine, for example PeSTO with PSQT only, or OliThink with mobility eval only.

I use to run STS1-15 for my Zeta engines (~2000 CCRL Elo), my development steps are still big enough to get a feeling of the progress via test suites:

Re: STS re-re-re-re-re-visited
https://talkchess.com/forum3/viewtopic. ... 10#p945710

E.g.: run STS to get a rough impression of your development, then measure Elo gain/loss via self-play, then measure Elo against other opponents.

I would say your search depth with given NPS and feature set is okay, you might want to take a look at your effective branching factor:

https://www.chessprogramming.org/Branch ... ing_Factor

Chess has an average game branching factor of ~36, with perfect AB move ordering you get an EBF of ~6, every additional (pruning) technique lowers EBF further, for example <2 for modern top AB engines.

Can not judge about your eval.

--
Srdja

[hgm]

I have never seen anyone trying to rate his engine based on the 'accuracy' reproted by chess.com.

It seems much better to just play a few games against other engines with known CCRL ratings.

[dangi12012]

I have never seen anyone trying to rate his engine based on the 'accuracy' reproted by chess.com.

It seems much better to just play a few games against other engines with known CCRL ratings.

This is the first thing to do imo.