The graphs have labels “importance of parameter”, so I intuited that to mean the process was zeroing in on where to focus, so to speak. Floundering a bit for words because it’s not clear how or if.Joerg Oster wrote: ↑Wed Sep 16, 2020 8:33 pmIf I understand it correctly, Optuna will eventually do both and also a mixture of both, to find out about the correlation of the parameters.chrisw wrote: ↑Wed Sep 16, 2020 5:30 pmYup, pretty much what I meantJoerg Oster wrote: ↑Wed Sep 16, 2020 4:30 pmI guess he wants to know if the tuner changes all parameters at once or one by one for a new trial.
In the document they refer to this as relational sampling and independent sampling.
Yet another parameter tuner using optuna framework
Moderators: hgm, Rebel, chrisw
-
- Posts: 4319
- Joined: Tue Apr 03, 2012 4:28 pm
Re: Yet another parameter tuner using optuna framework
-
- Posts: 4833
- Joined: Sun Aug 10, 2008 3:15 pm
- Location: Philippines
Re: Yet another parameter tuner using optuna framework
We can ask the optimizer either one by one or all at once. I did all at once. https://github.com/fsmosca/Optuna-Game- ... ner.py#L65Joerg Oster wrote: ↑Wed Sep 16, 2020 4:30 pmI guess he wants to know if the tuner changes all parameters at once or one by one for a new trial.
In the document they refer to this as relational sampling and independent sampling.
Example.
Code: Select all
pawn_value = trial.suggest_int('pawn_value', 50, 150, 2)
If I want more before making a trial run, I can ask for rook value for example.
Code: Select all
rook_value = trial.suggest_int('rook_value', 400, 600, 4)
-
- Posts: 4833
- Joined: Sun Aug 10, 2008 3:15 pm
- Location: Philippines
Re: Yet another parameter tuner using optuna framework
There can be parameter that does need too much games per trial. 24 can be high or low. There can be other conditions, like time control used, the number of parameters being tuned and the threshold of best value. To save optimization time one can start at lower number games. But what matter is after the trials the parameter values suggested by the optimizer improves over the default values.Joerg Oster wrote: ↑Wed Sep 16, 2020 4:53 pmThe number of games per trial is probably dependent on the sensitivity of the parameters, no?
24 games seems very small in any case.
-
- Posts: 4833
- Joined: Sun Aug 10, 2008 3:15 pm
- Location: Philippines
Re: Yet another parameter tuner using optuna framework
That can be tried for sure. I have not yet looked at the code inside the optimizer. It might already have a gradient calculated and automatic learning rate adjusted depending on the result of the match that we are sending to it after every trial.chrisw wrote: ↑Wed Sep 16, 2020 5:41 pm Looks very good. I still need to wrap my head around the graphs, will try later.
One thing, from your Github (and I don't want to flood you with suggested mods, I know how irritating that can be):
I tried something like this in the past (except with random kicks to the parameters, not smart ones as you are doing with Optima), where the parameters, P1, give a better result than P0 - was to move P fractionally towards P1, rather than the full thing.Second in order for the parameter values to be considered the best and replace the old best, it has to defeat the old best by more than 0.55 or 55% score. Normally this is only 0.5 or 50%.
This could work in the case of the close 50-55% range. Just an idea, but I expect you are full of ideas already!
-
- Posts: 4833
- Joined: Sun Aug 10, 2008 3:15 pm
- Location: Philippines
Re: Yet another parameter tuner using optuna framework
Not at the moment (will add it later), you need to modify the code around here https://github.com/fsmosca/Optuna-Game- ... er.py#L197
Correct.And I need only one copy of the engine in the folder, tuner just will execute two copies of it with a different parameters set?
The engine can be anywhere you can also specify an absolute path.
Code: Select all
python tuner.py --engine c:/chess/engines/enginefoldername/engineexefilename ...
-
- Posts: 29
- Joined: Tue Feb 02, 2010 10:12 pm
- Location: Germany
- Full name: Karlson Pfannschmidt
Re: Yet another parameter tuner using optuna framework
Thanks Ferdy for making your tool available! I like some of the plots you output for the results.
I wanted to point out that Optuna uses tree-structured parzen estimators as their model, which does not model interactions between parameters. Tools based on Gaussian processes (like the chess-tuning-tools or the tool released by thomasahle) take all interactions into account and thus are able to interpolate/extrapolate much more accurately.
I wanted to point out that Optuna uses tree-structured parzen estimators as their model, which does not model interactions between parameters. Tools based on Gaussian processes (like the chess-tuning-tools or the tool released by thomasahle) take all interactions into account and thus are able to interpolate/extrapolate much more accurately.
-
- Posts: 4833
- Joined: Sun Aug 10, 2008 3:15 pm
- Location: Philippines
Re: Yet another parameter tuner using optuna framework
Thanks for the info. The one with thomasahle takes a lot of memory tried it before. I have not yet tried your chess tuning tools. Will try it someday and compare it with optuna.Kiudee wrote: ↑Thu Sep 17, 2020 1:01 am Thanks Ferdy for making your tool available! I like some of the plots you output for the results.
I wanted to point out that Optuna uses tree-structured parzen estimators as their model, which does not model interactions between parameters. Tools based on Gaussian processes (like the chess-tuning-tools or the tool released by thomasahle) take all interactions into account and thus are able to interpolate/extrapolate much more accurately.
-
- Posts: 1784
- Joined: Wed Jul 03, 2019 4:42 pm
- Location: Netherlands
- Full name: Marcel Vanthoor
-
- Posts: 938
- Joined: Fri Mar 10, 2006 4:29 pm
- Location: Germany
Re: Yet another parameter tuner using optuna framework
Nevergrad might also be an interesting alternative.Ferdy wrote: ↑Thu Sep 17, 2020 1:07 amThanks for the info. The one with thomasahle takes a lot of memory tried it before. I have not yet tried your chess tuning tools. Will try it someday and compare it with optuna.Kiudee wrote: ↑Thu Sep 17, 2020 1:01 am Thanks Ferdy for making your tool available! I like some of the plots you output for the results.
I wanted to point out that Optuna uses tree-structured parzen estimators as their model, which does not model interactions between parameters. Tools based on Gaussian processes (like the chess-tuning-tools or the tool released by thomasahle) take all interactions into account and thus are able to interpolate/extrapolate much more accurately.
It offers a wide variety of optimization methods,
and has a nice ask and tell interface.
Jörg Oster
-
- Posts: 4367
- Joined: Fri Mar 10, 2006 5:23 am
- Location: http://www.arasanchess.org
Re: Yet another parameter tuner using optuna framework
Interesting. I also found this software:
https://github.com/automl/HpBandSter
which seems kind of similar. I have tried this sort of thing before especially for search parameters, but the problem I've found is that the effect of varying these can be quite small. So you are trying to find the optimum point but basically on a very "flat" surface, and furthermore with a method that produces noisy objective measures. It is therefore hard to get convergence.
https://github.com/automl/HpBandSter
which seems kind of similar. I have tried this sort of thing before especially for search parameters, but the problem I've found is that the effect of varying these can be quite small. So you are trying to find the optimum point but basically on a very "flat" surface, and furthermore with a method that produces noisy objective measures. It is therefore hard to get convergence.