txt: automated chess engine tuning

Discussion of chess software programming and technical issues.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
Post Reply
zamar
Posts: 613
Joined: Sun Jan 18, 2009 6:03 am

Re: txt: automated chess engine tuning

Post by zamar » Mon Mar 30, 2015 10:11 am

brtzsnr wrote:Hi, all!

I wrote a small framework to do automated chess tuning based on Texel's Tuning method (https://chessprogramming.wikispaces.com ... ing+Method). You can find the source code and instructions how to use it here https://bitbucket.org/brtzsnr/txt.

I'm still experimenting with it, so I cannot yet report any success. Nevertheless, feel free to experiment with txt and if you find it useful, please consider contributing.

Regards,
Years ago, I tried to do (almost) exactly what you describe there with Stockfish.

Even with a simple hill climb algorithm, the algorithm converged extremely fast into certain values.

Unfortunately the results were always a disaster in practical tests...
I think Komodo team has reported similar results.

But keep on trying. As always, the devil is in the details. So maybe someone can succeed in where I failed terribly.
Joona Kiiski

brtzsnr
Posts: 433
Joined: Fri Jan 16, 2015 3:02 pm
Contact:

Re: txt: automated chess engine tuning

Post by brtzsnr » Mon Mar 30, 2015 10:54 am

zamar wrote:
Years ago, I tried to do (almost) exactly what you describe there with Stockfish.

Even with a simple hill climb algorithm, the algorithm converged extremely fast into certain values.

Unfortunately the results were always a disaster in practical tests...
I think Komodo team has reported similar results.

But keep on trying. As always, the devil is in the details. So maybe someone can succeed in where I failed terribly.
Some questions if you remember:

1) How many positions did you use?
2) How did you select those positions?
3) What did you try to tune?
4) Did you select only a small variables, or tuned them all?

zamar
Posts: 613
Joined: Sun Jan 18, 2009 6:03 am

Re: txt: automated chess engine tuning

Post by zamar » Mon Mar 30, 2015 12:09 pm

brtzsnr wrote:
zamar wrote:
Years ago, I tried to do (almost) exactly what you describe there with Stockfish.

Even with a simple hill climb algorithm, the algorithm converged extremely fast into certain values.

Unfortunately the results were always a disaster in practical tests...
I think Komodo team has reported similar results.

But keep on trying. As always, the devil is in the details. So maybe someone can succeed in where I failed terribly.
Some questions if you remember:

1) How many positions did you use?
2) How did you select those positions?
3) What did you try to tune?
4) Did you select only a small variables, or tuned them all?
1) Can't remember exactly, but I think there were at least several hundred thousands, maybe a million or so.
2) I selected quiet positions from CCRL game database. My definition for quiet was: eval = qsearch(), also excluded positions in the very late endgame.
3) Material (first order, second order), mobility
4) I think I did a few runs for different combinations.
Joona Kiiski

zamar
Posts: 613
Joined: Sun Jan 18, 2009 6:03 am

Re: txt: automated chess engine tuning

Post by zamar » Mon Mar 30, 2015 2:23 pm

zamar wrote: 2) I selected quiet positions from CCRL game database. My definition for quiet was: eval = qsearch(), also excluded positions in the very late endgame.
I also made another attempt where I modified qsearch() to actually save fens of all quiet positions it encountered during the search with certain low probability. After running a few thousand games, I had a collection of several millions positions.

Then I tried to use these games/positions, but again it was of no good...
Joona Kiiski

Joerg Oster
Posts: 704
Joined: Fri Mar 10, 2006 3:29 pm
Location: Germany

Re: txt: automated chess engine tuning

Post by Joerg Oster » Tue Mar 31, 2015 3:56 am

Really odd!
All values are now at their maximum or minimum. :?:
Jörg Oster

brtzsnr
Posts: 433
Joined: Fri Jan 16, 2015 3:02 pm
Contact:

Re: txt: automated chess engine tuning

Post by brtzsnr » Tue Mar 31, 2015 8:49 am

Which values did you try to optimize? How common are they (e.g. white king on 8th rank in middle game is not very common)?

Joerg Oster
Posts: 704
Joined: Fri Mar 10, 2006 3:29 pm
Location: Germany

Re: txt: automated chess engine tuning

Post by Joerg Oster » Tue Mar 31, 2015 10:16 am

brtzsnr wrote:Which values did you try to optimize? How common are they (e.g. white king on 8th rank in middle game is not very common)?
It's the three rook parameters, midgame and endgame bonus.
From evaluate.cpp

Code: Select all

  // Assorted bonuses and penalties used by evaluation
... snip
  const Score RookOnPawn         = S( 7, 27);
  const Score RookOnOpenFile     = S(43, 21);
  const Score RookOnSemiOpenFile = S(19, 10);
RookOnPawn is especially helpful in the endgame, RookOnOpenFile and RookOnSemiOpenFile should be not that uncommon. :lol:

Maybe I chose too narrow range of values, +-10 from the default ones ...

I am aware that those parameters are already very well tuned, but I did hope this tuning method would be especially helpful in fine-tuning.
Jörg Oster

zamar
Posts: 613
Joined: Sun Jan 18, 2009 6:03 am

Re: txt: automated chess engine tuning

Post by zamar » Tue Mar 31, 2015 10:29 am

Joerg Oster wrote:Really odd!
All values are now at their maximum or minimum. :?:
That shouldn't happen... Sounds like a bug in tuner implementation...
Joona Kiiski

Henk
Posts: 6138
Joined: Mon May 27, 2013 8:31 am

Re: txt: automated chess engine tuning

Post by Henk » Tue Mar 31, 2015 11:48 am

Joerg Oster wrote:Really odd!
All values are now at their maximum or minimum. :?:
I've seen that more times when tuning other problems. It usually means that you have to wait much longer for you get more realistic values.

Joerg Oster
Posts: 704
Joined: Fri Mar 10, 2006 3:29 pm
Location: Germany

Re: txt: automated chess engine tuning

Post by Joerg Oster » Tue Mar 31, 2015 12:20 pm

zamar wrote:
Joerg Oster wrote:Really odd!
All values are now at their maximum or minimum. :?:
That shouldn't happen... Sounds like a bug in tuner implementation...
I think it's also possible that I must use wider bounds.
And for this kind of fine-tuning an even larger number of positions might also help.
Jörg Oster

Post Reply