Suggestions for a sparring partner

[AlvaroBegue]

I have been writing a new engine for about a month. Ideally I would like to use a sparring partner for my engine that is moderately stronger, because it's hard to learn what your biggest weaknesses are if you are matched against an overwhelmingly stronger opponent.

My new engine currently beats Fairy Max fairly systematically, it wins more than half of the games against gnuchess, but it losses badly to crafty or arasan.

Can anyone propose a good freely-available program that is likely to be stronger than my engine but not by too much? It is also important that it be able to handle very fast time controls.

Incidentally, I am using 10 seconds + 0.1 seconds per move Fisher clock for tests. What do others use?

[jdart]

You can adjust down Arasan's strength (use the UCI_LimitStrength option).

You could also try an older version. Version 11.7 for example is stable and is quite a bit weaker than current versions.

--Jon

[AlvaroBegue]

Thanks for the suggestion. I downloaded the Linux tar ball, ran `make' and `sudo make install', but it complains like this:

Error: required bitbase files not found.

[jdart]

bitbase files are in the data directory of the source distribution - copy to the place where your executable is.

--Jon

[lucasart]

I have been writing a new engine for about a month. Ideally I would like to use a sparring partner for my engine that is moderately stronger, because it's hard to learn what your biggest weaknesses are if you are matched against an overwhelmingly stronger opponent.

My new engine currently beats Fairy Max fairly systematically, it wins more than half of the games against gnuchess, but it losses badly to crafty or arasan.

Can anyone propose a good freely-available program that is likely to be stronger than my engine but not by too much? It is also important that it be able to handle very fast time controls.

Incidentally, I am using 10 seconds + 0.1 seconds per move Fisher clock for tests. What do others use?

Use self testing. And *do not* listen to all the religious dogma from Bob and Ed and the old guard of computer chess: they will tell you that self-testing is incest, and bla bla bla.

The fact is that self testing has practical advantages. In particular, for the same level of precision you need to play **4 times less** games in self-testing:
- 2x because of playing two gauntlets for the same nb of games per version
- and another 2x due to the "Pythagoric compounding" of error bars

PS: I also use 10"+0.1" for DiscoCheck. However, at the very early stages of developpement, you'll have tons of stupid bugs, and fixing them may not require much testing (just looking at a couple of games, and seeing that it's like day and night after the bugfix is sometimes enough, eg. if you inverted an inequality in the search, or used a ">" instead of >="etc.)

[Graham Banks]

Can anyone propose a good freely-available program that is likely to be stronger than my engine but not by too much? It is also important that it be able to handle very fast time controls.

Have a look at the rating lists.

[Evert]

Beware, however, that self testing doesn't help you identify holes in your evaluation function.

For instance: suppose that lack of understanding passed pawns is holding you back. If you realise this you can add it and the version that has it will beat the version that doesn't, but if you don't it's hard to figure out from looking at self play. However, an opponent that does have it will exploit and reveal this weakness.

Passed pawns is a fairly obvious evaluation term to have, but there may be others you don't think of. It certainly doesn't hurt to play against others programs and inspect the games to see if you can identify a common theme in why you lose. Just don't judge whether a change is an improvement just by looking at the games.

[lucasart]

Beware, however, that self testing doesn't help you identify holes in your evaluation function.

For instance: suppose that lack of understanding passed pawns is holding you back. If you realise this you can add it and the version that has it will beat the version that doesn't, but if you don't it's hard to figure out from looking at self play. However, an opponent that does have it will exploit and reveal this weakness.

Passed pawns is a fairly obvious evaluation term to have, but there may be others you don't think of. It certainly doesn't hurt to play against others programs and inspect the games to see if you can identify a common theme in why you lose. Just don't judge whether a change is an improvement just by looking at the games.

Yes, but you're mixing two completely different things:
(1) my point is that self-testing is the most convinient way to validate code patches
(2) your point is that it's also useful to look at games (ie. looking at a *few* games and analyzing them) in order to find ideas. These ideas have to be submitted to (1), however. I can't begin to count the number of "good ideas" that I've had by looking at games that ended up being regression. So if you do (2) w/o (1), you are doomed to fail: this is what programmers did in the old days, and that's why their programs are so crap by modern standards (compared to simple programs industrially engineered by applying (1) systematically)

So yes, you need good ideas, and good implementation (even more important than ideas), and good testing methodology (the most important).

[AlvaroBegue]

Beware, however, that self testing doesn't help you identify holes in your evaluation function.

For instance: suppose that lack of understanding passed pawns is holding you back. If you realise this you can add it and the version that has it will beat the version that doesn't, but if you don't it's hard to figure out from looking at self play. However, an opponent that does have it will exploit and reveal this weakness.

Passed pawns is a fairly obvious evaluation term to have, but there may be others you don't think of. It certainly doesn't hurt to play against others programs and inspect the games to see if you can identify a common theme in why you lose. Just don't judge whether a change is an improvement just by looking at the games.

That's my thinking exactly, and I want the sparring partner precisely to reveal those weaknesses in my evaluation function.

[AlvaroBegue]

Use self testing. And *do not* listen to all the religious dogma from Bob and Ed and the old guard of computer chess: they will tell you that self-testing is incest, and bla bla bla.

The fact is that self testing has practical advantages. In particular, for the same level of precision you need to play **4 times less** games in self-testing:
- 2x because of playing two gauntlets for the same nb of games per version
- and another 2x due to the "Pythagoric compounding" of error bars

I know. Actually, I explained that in a recent thread when Bob asked where the factor of 4 was coming from.

PS: I also use 10"+0.1" for DiscoCheck. However, at the very early stages of developpement, you'll have tons of stupid bugs, and fixing them may not require much testing (just looking at a couple of games, and seeing that it's like day and night after the bugfix is sometimes enough, eg. if you inverted an inequality in the search, or used a ">" instead of >="etc.)

I think I am past the "tons of stupid bugs" stage. Or perhaps I am too stupid to see them.