Yup, check. I was really quite surprised at just how dull search parameters proved to be, it's almost as if 'doing' a process is the most important bit and attempts to fine tune relatively fruitless. Compared to eval tuning where results plus/minus are semi-immediate and changes have strong effects. But search is not like eval, eval is grey scale accumulative, search is full of binary decision points (do I prune or not) and although the binary can be ameliorated to some extent (by adjusting how much depth left to search and so on, rather than yes/no cut/don't) there are still plenty of points where decision is full on binary, and a number that works for one case won't work for another etc. etc. Difficult problem.jdart wrote: ↑Thu Sep 17, 2020 4:47 pm Interesting. I also found this software:
https://github.com/automl/HpBandSter
which seems kind of similar. I have tried this sort of thing before especially for search parameters, but the problem I've found is that the effect of varying these can be quite small. So you are trying to find the optimum point but basically on a very "flat" surface, and furthermore with a method that produces noisy objective measures. It is therefore hard to get convergence.
Yet another parameter tuner using optuna framework
Moderators: hgm, Rebel, chrisw
-
- Posts: 4317
- Joined: Tue Apr 03, 2012 4:28 pm
Re: Yet another parameter tuner using optuna framework
-
- Posts: 937
- Joined: Fri Mar 10, 2006 4:29 pm
- Location: Germany
Re: Yet another parameter tuner using optuna framework
Which seems to be the most reasonable, imho.Ferdy wrote: ↑Thu Sep 17, 2020 12:05 amWe can ask the optimizer either one by one or all at once. I did all at once. https://github.com/fsmosca/Optuna-Game- ... ner.py#L65Joerg Oster wrote: ↑Wed Sep 16, 2020 4:30 pmI guess he wants to know if the tuner changes all parameters at once or one by one for a new trial.
In the document they refer to this as relational sampling and independent sampling.
Jörg Oster
-
- Posts: 105
- Joined: Thu Jun 18, 2020 3:21 pm
- Location: Moscow
- Full name: Alexander Litov
Re: Yet another parameter tuner using optuna framework
I mean should the chess engine itself (one that i want to tune) obtain parameters via command line fe as "QueenValueOp=975" ?Ferdy wrote: ↑Thu Sep 17, 2020 12:53 amNot at the moment (will add it later), you need to modify the code around here https://github.com/fsmosca/Optuna-Game- ... er.py#L197
I want to know what to implement to work with your tool.
It is quite clear from your info that i should just write needed params in the tuner file itself, my question is how it (the tuner) communicate this params to the engine I want to tune. I tried to looked into the tuner code and my best guess is that tuner will run engine with smth like
Code: Select all
engine.exe "QueenValueOp=975"
Sorry if it is obvious and i`m being stupid. Im not that good at coding.
-
- Posts: 4833
- Joined: Sun Aug 10, 2008 3:15 pm
- Location: Philippines
Re: Yet another parameter tuner using optuna framework
I use cutechess-cli to set the suggested params to the engine.No4b wrote: ↑Thu Sep 17, 2020 11:21 pmI mean should the chess engine itself (one that i want to tune) obtain parameters via command line fe as "QueenValueOp=975" ?Ferdy wrote: ↑Thu Sep 17, 2020 12:53 amNot at the moment (will add it later), you need to modify the code around here https://github.com/fsmosca/Optuna-Game- ... er.py#L197
I want to know what to implement to work with your tool.
It is quite clear from your info that i should just write needed params in the tuner file itself, my question is how it (the tuner) communicate this params to the engine I want to tune.
Once I get the suggested param from the optimizer, I build a text string containing,
Code: Select all
option.PawnValueEn=92 option.BishopValueOp=324
Code: Select all
cutechess-cli.exe -engine cmd=deuterium.exe proto=uci name=deuterium_piecevalue option.PawnValueEn=92 option.BishopValueOp=324 -engine cmd=deuterium.exe proto=uci name=deuterium_default ...
-
- Posts: 4833
- Joined: Sun Aug 10, 2008 3:15 pm
- Location: Philippines
Re: Yet another parameter tuner using optuna framework
@Joerg
Could you create a Stockfish where it exposes its material piece value and other evaluation param perhaps and a couple of search parameters. I can make it but may take sometime. I want to test optuna with it and later compare with other tuner like spsa and chess tuning tools and others perhaps. Stockfish params are well optimized, want to see how many elo optuna can increase or decrease. Or unoptimize it and try to optimize it back. Don't put a space in the uci option name and build an exe for modern. You can use the one with nnue but I prefer the one without it. Even a revised Stockfish 11 would do. Thanks.
Could you create a Stockfish where it exposes its material piece value and other evaluation param perhaps and a couple of search parameters. I can make it but may take sometime. I want to test optuna with it and later compare with other tuner like spsa and chess tuning tools and others perhaps. Stockfish params are well optimized, want to see how many elo optuna can increase or decrease. Or unoptimize it and try to optimize it back. Don't put a space in the uci option name and build an exe for modern. You can use the one with nnue but I prefer the one without it. Even a revised Stockfish 11 would do. Thanks.
-
- Posts: 937
- Joined: Fri Mar 10, 2006 4:29 pm
- Location: Germany
Re: Yet another parameter tuner using optuna framework
Hi, I also intended to run some Optuna tuning sessions with Stockfish.Ferdy wrote: ↑Fri Sep 18, 2020 1:25 am @Joerg
Could you create a Stockfish where it exposes its material piece value and other evaluation param perhaps and a couple of search parameters. I can make it but may take sometime. I want to test optuna with it and later compare with other tuner like spsa and chess tuning tools and others perhaps. Stockfish params are well optimized, want to see how many elo optuna can increase or decrease. Or unoptimize it and try to optimize it back. Don't put a space in the uci option name and build an exe for modern. You can use the one with nnue but I prefer the one without it. Even a revised Stockfish 11 would do. Thanks.
There were some SPSA sessions running in fishtest for the mobility values recently.
I'm also very interested if Optuna can handle high-dimensional problems.
Not sure if I manage to get the piece values exposed for tuning, but some search and eval parameters for sure.
Not sure if I get it done today, though.
As for the number of games to be played per trial, I think I have to step back from my previous statement.
It seems more important to get more trials done in a reasonable amount of time, than to (further) decrease the noise of the measurements.
Even when playing 512 games per round/trial, there would still be much noise involved.
Jörg Oster
-
- Posts: 4833
- Joined: Sun Aug 10, 2008 3:15 pm
- Location: Philippines
Re: Yet another parameter tuner using optuna framework
Looking forward on your results.Joerg Oster wrote: ↑Fri Sep 18, 2020 12:19 pmHi, I also intended to run some Optuna tuning sessions with Stockfish.Ferdy wrote: ↑Fri Sep 18, 2020 1:25 am @Joerg
Could you create a Stockfish where it exposes its material piece value and other evaluation param perhaps and a couple of search parameters. I can make it but may take sometime. I want to test optuna with it and later compare with other tuner like spsa and chess tuning tools and others perhaps. Stockfish params are well optimized, want to see how many elo optuna can increase or decrease. Or unoptimize it and try to optimize it back. Don't put a space in the uci option name and build an exe for modern. You can use the one with nnue but I prefer the one without it. Even a revised Stockfish 11 would do. Thanks.
There were some SPSA sessions running in fishtest for the mobility values recently.
I'm also very interested if Optuna can handle high-dimensional problems.
No problem, I will take what you will have. Sending it next month is fine with me.Not sure if I manage to get the piece values exposed for tuning, but some search and eval parameters for sure.
Not sure if I get it done today, though.
This tuning is getting fun while it consumes lots of resources too.As for the number of games to be played per trial, I think I have to step back from my previous statement.
It seems more important to get more trials done in a reasonable amount of time, than to (further) decrease the noise of the measurements.
Even when playing 512 games per round/trial, there would still be much noise involved.
In one of my studies, I looked at the importances plot - if value is high that parameter has more influence on the match result.
Those are the bishops.
Now if we look at the slice plot, the tuner is somewhat confused, the trials are all over the place for the BishopValueEn. Trials are somewhat divided in the 320 to 340 range and also in the 340 to 360 range. The number of trials for those plots are 300 from 24 games/trial at tc=15s+100ms. In this situation I can continue the trials to perhaps another 100 trials and see what happens to importances and slice plot. After 300 trials I get around +12 wins in favor of the tuned param values over the default, at around 800 games in the actual game match at tc=15s+100ms. I discontinued this study.
It is also showing in the contour plot for BishopValueEn. Majority of trials are in the range 320 to 340 and 340 to 360.
-
- Posts: 937
- Joined: Fri Mar 10, 2006 4:29 pm
- Location: Germany
Re: Yet another parameter tuner using optuna framework
@FerdyJoerg Oster wrote: ↑Fri Sep 18, 2020 12:19 pmHi, I also intended to run some Optuna tuning sessions with Stockfish.Ferdy wrote: ↑Fri Sep 18, 2020 1:25 am @Joerg
Could you create a Stockfish where it exposes its material piece value and other evaluation param perhaps and a couple of search parameters. I can make it but may take sometime. I want to test optuna with it and later compare with other tuner like spsa and chess tuning tools and others perhaps. Stockfish params are well optimized, want to see how many elo optuna can increase or decrease. Or unoptimize it and try to optimize it back. Don't put a space in the uci option name and build an exe for modern. You can use the one with nnue but I prefer the one without it. Even a revised Stockfish 11 would do. Thanks.
There were some SPSA sessions running in fishtest for the mobility values recently.
I'm also very interested if Optuna can handle high-dimensional problems.
Not sure if I manage to get the piece values exposed for tuning, but some search and eval parameters for sure.
Not sure if I get it done today, though.
As for the number of games to be played per trial, I think I have to step back from my previous statement.
It seems more important to get more trials done in a reasonable amount of time, than to (further) decrease the noise of the measurements.
Even when playing 512 games per round/trial, there would still be much noise involved.
I've put something online here https://github.com/joergoster/Stockfish/releases/tag/v1
It's a modern build from end of July version, pre NNUE, with some eval parameters made tunable, plus 2 search margins.
Hope it works for you.
If you want some other parameters exposed for tuning, please let me know.
Jörg Oster
-
- Posts: 4833
- Joined: Sun Aug 10, 2008 3:15 pm
- Location: Philippines
Re: Yet another parameter tuner using optuna framework
I got the file thanks.Joerg Oster wrote: ↑Sat Sep 19, 2020 9:14 am @Ferdy
I've put something online here https://github.com/joergoster/Stockfish/releases/tag/v1
It's a modern build from end of July version, pre NNUE, with some eval parameters made tunable, plus 2 search margins.
Hope it works for you.
If you want some other parameters exposed for tuning, please let me know.
-
- Posts: 2801
- Joined: Mon Feb 11, 2008 3:53 pm
- Location: Denmark
- Full name: Damir Desevac
Re: Yet another parameter tuner using optuna framework
Hi Jörg
Thanks a lot for this tunable Stockfish. If possible can you also put NNUE inside, ( whitout) embeeded net
Thanks a lot for this tunable Stockfish. If possible can you also put NNUE inside, ( whitout) embeeded net