Quantifying progress && Naked WAC

[AxolotlFever]

Hi all,

I am trying to make my engine better, and it can be hard to tell if it is weak because it has bugs, or just because the evaluation function is not so good and everything is slow.

First, I would be really grateful if anyone can give me ideas of how to quantify progress. I could connect to arena and play other engines, but 1. they are all too strong for me and 2. this takes a while to see results.

Currently I am using the WAC test suite, at 20 seconds a move, and I get about 150 / 300, without null move / tt / lmr. With everything turned on, I get to about 170. Has anyone else compared their regular WAC scores with a "naked" WAC ?

Kind regards, and thanks for any help,
Louis

PS: I would in particular find it really useful if you could say something like "at 30 seconds a move, with only basic alpha beta, you should get a score of xxx / 300". This would give me a clear way of evaluating where my engine is failing. Thanks!

[xr_a_y]

you can play 1000 games at 0:10/40 or 0:20/40 TC against tscp or micro-max.

[Ratosh]

Hi Louis,

If you wanna test the base strength of your engine you can use some really low rated engine on CCRL or any other rating list, but the best way to measure progress is using SPRT. I use OpenBench by Andrew Grant, it has integration with GitHub branches, so you can test if a branch is "better" than development, It's pretty similar to Fish-test but you run the server and the client on your machine. On my OpenBench fork you can see the changes i made on the client to build my engine using Gradle.

PS.: The link is pointing to my fork.

[Dann Corbit]

Hi all,

I am trying to make my engine better, and it can be hard to tell if it is weak because it has bugs, or just because the evaluation function is not so good and everything is slow.

First, I would be really grateful if anyone can give me ideas of how to quantify progress. I could connect to arena and play other engines, but 1. they are all too strong for me and 2. this takes a while to see results.

Currently I am using the WAC test suite, at 20 seconds a move, and I get about 150 / 300, without null move / tt / lmr. With everything turned on, I get to about 170. Has anyone else compared their regular WAC scores with a "naked" WAC ?

Kind regards, and thanks for any help,
Louis

PS: I would in particular find it really useful if you could say something like "at 30 seconds a move, with only basic alpha beta, you should get a score of xxx / 300". This would give me a clear way of evaluating where my engine is failing. Thanks!

The absolute score of the WAC is probably not that important.
What it is most useful for is to find holes in evaluation.
If it is a king safety position and you do not find it, then look at your king safety.

Another fun test is to permute the WAC and see if you get the same results (you should).
You might see minor differences due to SMP or if you order the moves differently.

[xr_a_y]

Is there a formula with WAC to convert results xxx/300 to estimated elo ? I know some other test suite does provide this, maybe someone has one for WAC ?

[hgm]

That doesn't seem possible. It used to be that tuning your engine for maximum score would actually decrease its Elo. (Not sure whether this is still the case for modern engines.) As extremely weak engines (e.g. with hardy any search depth) would also not score very well, that would mean any relation between WAC score and Elo would not be a 1:1 mapping.

[JVMerlino]

That doesn't seem possible. It used to be that tuning your engine for maximum score would actually decrease its Elo. (Not sure whether this is still the case for modern engines.) As extremely weak engines (e.g. with hardy any search depth) would also not score very well, that would mean any relation between WAC score and Elo would not be a 1:1 mapping.

While I (mostly) agree with this, scoring only 170/300 at 20 seconds per move shows some clear problems. Myrddin v0.82, which was rated around 1925, could get 262/300 at 10 seconds per move. That's on hardware that would be 10+ years old now.

I also highly doubt that the problem is in the evaluation (as the OP suggested this as one of the possible issues). I would be curious to see some output from the engine showing nodes per second and time to depth. Or even the entire output from your WAC test. This could enlighten us quite a bit.

[xr_a_y]

For what it's worth Minic 0.28 is scoring 266/300 @ 3sec per position on a 7 years old hardware single thread.

[Dann Corbit]

Is there a formula with WAC to convert results xxx/300 to estimated elo ? I know some other test suite does provide this, maybe someone has one for WAC ?

The main purpose for a simple test like WAC is to find programming errors and missing evaluation features.

You can tune your engine to WAC in the early stages of writing it, but for best results in game play you should tune it with game play.

If you get many wrong answers with WAC problems at 10 seconds per position, then you have errors in your evaluation and search.

A modern engine should easily get 275/300 at that time control. There are many engines which will get all but WAC.230 correct at ten seconds per move. WAC.230 is also busted, in that the best move does not win (proved by Alex Szabo) because there is an equally brilliant refutation where the defender creates a passed pawn. Still, the rook sacrifice is the only move that has any winning chances, so in that sense it really is the best move.