txt: automated chess engine tuning

[zamar]

Hi, all!

I wrote a small framework to do automated chess tuning based on Texel's Tuning method (https://chessprogramming.wikispaces.com ... ing+Method). You can find the source code and instructions how to use it here https://bitbucket.org/brtzsnr/txt.

I'm still experimenting with it, so I cannot yet report any success. Nevertheless, feel free to experiment with txt and if you find it useful, please consider contributing.

Regards,

Years ago, I tried to do (almost) exactly what you describe there with Stockfish.

Even with a simple hill climb algorithm, the algorithm converged extremely fast into certain values.

Unfortunately the results were always a disaster in practical tests...
I think Komodo team has reported similar results.

But keep on trying. As always, the devil is in the details. So maybe someone can succeed in where I failed terribly.

[brtzsnr]

Years ago, I tried to do (almost) exactly what you describe there with Stockfish.

Even with a simple hill climb algorithm, the algorithm converged extremely fast into certain values.

Unfortunately the results were always a disaster in practical tests...
I think Komodo team has reported similar results.

But keep on trying. As always, the devil is in the details. So maybe someone can succeed in where I failed terribly.

Some questions if you remember:

1) How many positions did you use?
2) How did you select those positions?
3) What did you try to tune?
4) Did you select only a small variables, or tuned them all?

[zamar]

Years ago, I tried to do (almost) exactly what you describe there with Stockfish.

Even with a simple hill climb algorithm, the algorithm converged extremely fast into certain values.

Unfortunately the results were always a disaster in practical tests...
I think Komodo team has reported similar results.

But keep on trying. As always, the devil is in the details. So maybe someone can succeed in where I failed terribly.

Some questions if you remember:

1) How many positions did you use?
2) How did you select those positions?
3) What did you try to tune?
4) Did you select only a small variables, or tuned them all?

1) Can't remember exactly, but I think there were at least several hundred thousands, maybe a million or so.
2) I selected quiet positions from CCRL game database. My definition for quiet was: eval = qsearch(), also excluded positions in the very late endgame.
3) Material (first order, second order), mobility
4) I think I did a few runs for different combinations.

[zamar]

2) I selected quiet positions from CCRL game database. My definition for quiet was: eval = qsearch(), also excluded positions in the very late endgame.

I also made another attempt where I modified qsearch() to actually save fens of all quiet positions it encountered during the search with certain low probability. After running a few thousand games, I had a collection of several millions positions.

Then I tried to use these games/positions, but again it was of no good...

[Joerg Oster]

Really odd!
All values are now at their maximum or minimum.

[brtzsnr]

Which values did you try to optimize? How common are they (e.g. white king on 8th rank in middle game is not very common)?

[Joerg Oster]

Which values did you try to optimize? How common are they (e.g. white king on 8th rank in middle game is not very common)?

It's the three rook parameters, midgame and endgame bonus.
From evaluate.cpp

Code: Select all

  // Assorted bonuses and penalties used by evaluation
... snip
  const Score RookOnPawn         = S( 7, 27);
  const Score RookOnOpenFile     = S(43, 21);
  const Score RookOnSemiOpenFile = S(19, 10);

RookOnPawn is especially helpful in the endgame, RookOnOpenFile and RookOnSemiOpenFile should be not that uncommon.

Maybe I chose too narrow range of values, +-10 from the default ones ...

I am aware that those parameters are already very well tuned, but I did hope this tuning method would be especially helpful in fine-tuning.

[zamar]

Really odd!
All values are now at their maximum or minimum.

That shouldn't happen... Sounds like a bug in tuner implementation...

[Henk]

Really odd!
All values are now at their maximum or minimum.

I've seen that more times when tuning other problems. It usually means that you have to wait much longer for you get more realistic values.

[Joerg Oster]

Really odd!
All values are now at their maximum or minimum.

That shouldn't happen... Sounds like a bug in tuner implementation...

I think it's also possible that I must use wider bounds.
And for this kind of fine-tuning an even larger number of positions might also help.