Stockfish's tuning method

Discussion of chess software programming and technical issues.

Moderators: hgm, Harvey Williamson, bob

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
Rémi Coulom
Posts: 426
Joined: Mon Apr 24, 2006 6:06 pm
Contact:

Re: Stockfish's tuning method

Post by Rémi Coulom » Fri Oct 07, 2011 7:29 pm

mcostalba wrote:The name "ampli-bias knobs" was chosen as an analogy to old analog TV where you can control the full image view (made of many pixels) using only two knobs: contrast (ampli) and luminance (bias).
Thanks. I understand now.

Rémi

jdart
Posts: 3754
Joined: Fri Mar 10, 2006 4:23 am
Location: http://www.arasanchess.org

Re: Stockfish's tuning method

Post by jdart » Sat Oct 08, 2011 2:06 am

How many CPUs, and how much time, do you need for 30,000 games?

mjlef
Posts: 1402
Joined: Thu Mar 30, 2006 12:08 pm
Contact:

Re: Stockfish's tuning method

Post by mjlef » Sat Oct 08, 2011 2:30 am

and were these fixed depth or timed matches?

mcostalba
Posts: 2684
Joined: Sat Jun 14, 2008 7:17 pm

Re: Stockfish's tuning method

Post by mcostalba » Sat Oct 08, 2011 2:50 am

Rémi Coulom wrote: Thanks. I understand now.
My mathematics skills are very rusty these days, but more or less (forgive me if I am not precise), if we consider the N parameters to tune in parallel a vector of dimension N, then we tune instead the coefficients of the matrix for which the vector is an eigenvector.

zamar
Posts: 613
Joined: Sun Jan 18, 2009 6:03 am

Re: Stockfish's tuning method

Post by zamar » Sat Oct 08, 2011 2:12 pm

zamar wrote:The method is a practical approach and not mathematically very sound. Because algorithm is very simple, it's very
likely already invented a long time ago.
It is not so unsound. It is like the SPSA algorithm, except SPSA does not use self-play. You can read about SPSA there, if you are interested:
http://www.jhuapl.edu/SPSA/
Thanks for the link Remi. Yes, it looks like that our method is SPSA with the exception of self-play.
As I mentioned in my paper, SPSA has the potential to be close in performance to CLOP, but its main weakness (as Joona says) is that it is very difficult to choose good values for all its meta-parameters. In my experiments, SPSA with optimal meta-parameters performs like CLOP. But in practice, it is not possible to find the optimal meta-parameters of SPSA, so I'd prefer using CLOP.
The attractiveness in SPSA for us is that when we already have a very good starting value, it immediately starts to improve it. While CLOP (if I've understood it correctly) always starts from scratch. Most of the SF tuning was done using only one QUAD-core computer, so we could only use at maximum 100'000 games for each set of variables (set containing 7-30 variables).

But I don't know, it's of course possible that CLOP could have done better...
Joona Kiiski

zamar
Posts: 613
Joined: Sun Jan 18, 2009 6:03 am

Re: Stockfish's tuning method

Post by zamar » Sat Oct 08, 2011 2:17 pm

jdart wrote:How many CPUs, and how much time, do you need for 30,000 games?
We used time controls to reach 100'000 games/1 week. Only one quad core computer was used. We ran 1CPU matches on parallel although this adds some extra noise to the results.
Joona Kiiski

zamar
Posts: 613
Joined: Sun Jan 18, 2009 6:03 am

Re: Stockfish's tuning method

Post by zamar » Sat Oct 08, 2011 2:25 pm

mjlef wrote:and were these fixed depth or timed matches?
We used time controls 5s+0.1s, 10s+0.1s and 20s+0.1s. If we suspected that variables are dependant on the time control, we used longer time controls and ran fewer matches on parallel.
Joona Kiiski

ethanara
Posts: 134
Joined: Mon May 16, 2011 4:58 pm
Location: Denmark
Contact:

Re: Stockfish's tuning method

Post by ethanara » Sat Oct 08, 2011 2:33 pm

Can you tell which variables are tuned in stockfish? Or if there are more tuned than untuned, then which are untuned?
Regards
Ethan

zamar
Posts: 613
Joined: Sun Jan 18, 2009 6:03 am

Re: Stockfish's tuning method

Post by zamar » Sat Oct 08, 2011 2:39 pm

Gerd Isenberg wrote:
zamar wrote:Created quickly a page in wiki:

https://chessprogramming.wikispaces.com ... ing+method
Oups sorry, I didn't recognize it was you (not aware of all you confusing alias names), but thought somebody else had copied/pasted the text without quoting original source - so I deleted the page but restored it later ;-)

I have edited it slightly. Thank you!

Cheers,
Gerd
Thanks for cleaning up the page and linking it to the right places, Gerd!

Cheers,
Joona
Joona Kiiski

mcostalba
Posts: 2684
Joined: Sat Jun 14, 2008 7:17 pm

Re: Stockfish's tuning method

Post by mcostalba » Sat Oct 08, 2011 2:54 pm

ethanara wrote:Can you tell which variables are tuned in stockfish? Or if there are more tuned than untuned, then which are untuned?
Regards
Ethan
Because you are a little kid (12 year's old if I guess right) I take the chance to do a small game with you (no offence, just for playing), because I think it is important you learn early on to understand what your questions means (and this is important not only related to chess engines). This is a prerequisite to understand the corresponding answers.

So please, as a little homework, if you wish write here below the list of what you think can be "tunable variables" in SF and I will tell to you for each one of them if are already tuned of not. Just to be clear I mean the real variable names used in the program, not general, grouping names.

Post Reply