Laskas parameter optimizer

[Patrice Duhamel]

Right I see what you mean now.

Thanks for your help.
So if I let it end without interruption, it should print the best parameters ? (in my example the parameters from the first run ?)

I forgot to say that I interrupt it manually because I don't know how much time I can let it running.

This is my command line : (I put only the first 4 parameters because the line is long with 16 parameters)

Code: Select all

python lakas.py --concurrency 3 --hash 256 --optimizer bayesopt --bo-utility-kind ucb --input-data-file dataoptin.dat --output-data-file dataoptout.dat --optimizer-log-file opt_log_plot_ucb.txt --base-time-sec 10 --inc-time-sec 0.1 --budget 100 --games-per-budget 200 --engine ./engines/cheese/cheese-30-win-64-pop.exe --input-param "{'MobScaleKnightMid': {'init':48, 'lower':8, 'upper':80},'MobScaleKnightEnd': {'init':40, 'lower':8, 'upper':80},'MobTranslateKnightMid': {'init':0, 'lower':0, 'upper':80},'MobTranslateKnightEnd': {'init':0, 'lower':0, 'upper':80}}" --opening-file ./start_opening/ogpt_chess_startpos.epd --opening-file-format epd

Note that before restarting Lakas I don't forget to copy dataoptout.dat => dataoptin.dat

[Ferdy]

Right I see what you mean now.

Thanks for your help.
So if I let it end without interruption, it should print the best parameters ?

Yes.

(in my example the parameters from the first run ?)

Perhaps depending on the optimizer used, as there is optimizer that recommends best param by considering other conditions. Like when you use TBPSA

https://facebookresearch.github.io/neve ... rizedTBPSA


Sometimes, I might not notice that the library of nevergrad has changes to how they return the best param. Regardless we will get the best param after the scheduled budgets are completed.

In the update that I am working, the best param are now correctly printed after the first run.

First run ends:

Code: Select all

2021-01-29 20:11:40,223 | INFO  | opt num_ask: 10

2021-01-29 20:11:40,225 | INFO  | budget: 10
2021-01-29 20:11:40,225 | INFO  | recommended param: {'FutilityMargin': 73, 'LmrFactor': 99, 'QsearchFutilityMargin': 97, 'WinEvalDepth': 7}
2021-01-29 20:11:40,229 | INFO  | best param: {'FutilityMargin': 40, 'LmrFactor': 100, 'QsearchFutilityMargin': 86, 'WinEvalDepth': 8}
2021-01-29 20:11:40,229 | INFO  | init param: {'FutilityMargin': 60, 'LmrFactor': 100, 'QsearchFutilityMargin': 100, 'WinEvalDepth': 7}
2021-01-29 20:11:40,230 | INFO  | recommended vs init
2021-01-29 20:12:32,286 | INFO  | actual result: 0.56250 @24 games, minimized result: 0.43750, point of view: recommended
2021-01-29 20:12:32,286 | INFO  | best loss: 0.375

2021-01-29 20:12:32,293 | INFO  | best_param: {'FutilityMargin': 40, 'LmrFactor': 100, 'QsearchFutilityMargin': 86, 'WinEvalDepth': 8}
2021-01-29 20:12:32,451 | INFO  | option.FutilityMargin=40 option.LmrFactor=100 option.QsearchFutilityMargin=86 option.WinEvalDepth=8

Second run starts:

Code: Select all

2021-01-29 20:22:20,146 | INFO  | total budget: 10
2021-01-29 20:22:20,148 | INFO  | games_per_budget: 24
2021-01-29 20:22:20,151 | INFO  | tuning match move control: base_time_sec: 2, inc_time_sec: 0.1, depth=1000
2021-01-29 20:22:20,153 | INFO  | parameter dimension: 4
2021-01-29 20:22:20,176 | INFO  | bayesopt previous num_ask: 10

2021-01-29 20:22:20,229 | INFO  | opt num_ask: 11

2021-01-29 20:22:20,231 | INFO  | budget: 1
2021-01-29 20:22:20,232 | INFO  | recommended param: {'FutilityMargin': 76, 'LmrFactor': 97, 'QsearchFutilityMargin': 101, 'WinEvalDepth': 2}
2021-01-29 20:22:20,233 | INFO  | best param: {'FutilityMargin': 40, 'LmrFactor': 100, 'QsearchFutilityMargin': 86, 'WinEvalDepth': 8}
2021-01-29 20:22:20,233 | INFO  | init param: {'FutilityMargin': 60, 'LmrFactor': 100, 'QsearchFutilityMargin': 100, 'WinEvalDepth': 7}
2021-01-29 20:22:20,234 | INFO  | recommended vs init
2021-01-29 20:23:16,772 | INFO  | actual result: 0.47917 @24 games, minimized result: 0.52083, point of view: recommended

2021-01-29 20:23:16,824 | INFO  | opt num_ask: 12

2021-01-29 20:23:16,826 | INFO  | budget: 2
2021-01-29 20:23:16,827 | INFO  | recommended param: {'FutilityMargin': 68, 'LmrFactor': 86, 'QsearchFutilityMargin': 84, 'WinEvalDepth': 7}
2021-01-29 20:23:16,828 | INFO  | best param: {'FutilityMargin': 40, 'LmrFactor': 100, 'QsearchFutilityMargin': 86, 'WinEvalDepth': 8}
2021-01-29 20:23:16,828 | INFO  | init param: {'FutilityMargin': 60, 'LmrFactor': 100, 'QsearchFutilityMargin': 100, 'WinEvalDepth': 7}
2021-01-29 20:23:16,830 | INFO  | recommended vs init
2021-01-29 20:24:27,238 | INFO  | actual result: 0.35417 @24 games, minimized result: 0.64583, point of view: recommended

2021-01-29 20:24:27,312 | INFO  | opt num_ask: 13

2021-01-29 20:24:27,314 | INFO  | budget: 3
2021-01-29 20:24:27,315 | INFO  | recommended param: {'FutilityMargin': 61, 'LmrFactor': 100, 'QsearchFutilityMargin': 62, 'WinEvalDepth': 5}
2021-01-29 20:24:27,316 | INFO  | best param: {'FutilityMargin': 40, 'LmrFactor': 100, 'QsearchFutilityMargin': 86, 'WinEvalDepth': 8}
2021-01-29 20:24:27,316 | INFO  | init param: {'FutilityMargin': 60, 'LmrFactor': 100, 'QsearchFutilityMargin': 100, 'WinEvalDepth': 7}
2021-01-29 20:24:27,317 | INFO  | recommended vs init

...

Will upload it after some tests.

Note that before restarting Lakas I don't forget to copy dataoptout.dat => dataoptin.dat

There is no need to copy (unless you want a backup of previous run) you can use the same filename like:

Code: Select all

--input-data-file dataopt.dat --output-data-file dataopt.dat

[Ferdy]

There is new update on nevergrad lib.

By default, the optimizer now returns the best set of parameter as recommendation #951, considering that the function is deterministic. The previous behavior would use an estimation of noise to provide the pessimistic best point, leading to unexpected behaviors #947. You can can back to this behavior by specifying: :code:parametrization.descriptors.deterministic_function = False

The best param returned is now based on the assumption that the objective is deterministic. I will have to make some changes in Lakas, possibly make it an option.

[Ferdy]

lakas.py v0.18.0 is now released.

Better output display when resuming an optimization session and budget number is updated continuously from session to session.

Last budget of session 4.

Code: Select all

2021-01-30 02:40:51,768 | INFO  | budget: 40
2021-01-30 02:40:51,768 | INFO  | recommended param: {'FutilityMargin': 44, 'LmrFactor': 94, 'QsearchFutilityMargin': 56, 'WinEvalDepth': 6}
2021-01-30 02:40:51,768 | INFO  | best param: {'FutilityMargin': 54, 'LmrFactor': 99, 'QsearchFutilityMargin': 132, 'WinEvalDepth': 6}
2021-01-30 02:40:51,769 | INFO  | best loss: 0.35416666666666663

2021-01-30 02:40:51,770 | INFO  | init param: {'FutilityMargin': 60, 'LmrFactor': 100, 'QsearchFutilityMargin': 100, 'WinEvalDepth': 7}
2021-01-30 02:40:51,770 | INFO  | recommended vs init
2021-01-30 02:42:07,354 | INFO  | actual result: 0.47917 @24 games, minimized result: 0.52083, point of view: recommended

2021-01-30 02:42:07,364 | INFO  | best_param: {'FutilityMargin': 54, 'LmrFactor': 99, 'QsearchFutilityMargin': 132, 'WinEvalDepth': 6}
2021-01-30 02:42:07,496 | INFO  | option.FutilityMargin=54 option.LmrFactor=99 option.QsearchFutilityMargin=132 option.WinEvalDepth=6

First budget of session 5.

Code: Select all

2021-01-30 02:46:11,742 | INFO  | total budget: 10
2021-01-30 02:46:11,744 | INFO  | games_per_budget: 24
2021-01-30 02:46:11,747 | INFO  | tuning match move control: base_time_sec: 2, inc_time_sec: 0.1, depth=1000
2021-01-30 02:46:11,749 | INFO  | parameter dimension: 4
2021-01-30 02:46:11,766 | INFO  | tbpsa previous num_ask: 40

2021-01-30 02:46:11,773 | INFO  | budget: 41
2021-01-30 02:46:11,773 | INFO  | recommended param: {'FutilityMargin': 80, 'LmrFactor': 89, 'QsearchFutilityMargin': 107, 'WinEvalDepth': 6}
2021-01-30 02:46:11,773 | INFO  | best param: {'FutilityMargin': 54, 'LmrFactor': 99, 'QsearchFutilityMargin': 132, 'WinEvalDepth': 6}
2021-01-30 02:46:11,774 | INFO  | best loss: 0.35416666666666663
2021-01-30 02:46:11,776 | INFO  | init param: {'FutilityMargin': 60, 'LmrFactor': 100, 'QsearchFutilityMargin': 100, 'WinEvalDepth': 7}
2021-01-30 02:46:11,776 | INFO  | recommended vs init
2021-01-30 02:47:10,519 | INFO  | actual result: 0.52083 @24 games, minimized result: 0.47917, point of view: recommended

On the html output, in the hiplot graph, session is the number of resuming optimization, num-tell is the budget, loss low is better with parameter values at right. This html output is updated after every session is completed.

[Patrice Duhamel]

lakas.py v0.18.0 is now released.

Thanks, I will try it, with more games.

[Ferdy]

lakas.py v0.18.0 is now released.

Thanks, I will try it, with more games.

There is lakas v0.21.0, this one uses the latest nevergrad v0.4.3.

Update your nevergrad with:

Code: Select all

pip install -U nevergrad

[Joerg Oster]

lakas.py v0.18.0 is now released.

Thanks, I will try it, with more games.

There is lakas v0.21.0, this one uses the latest nevergrad v0.4.3.

Update your nevergrad with:

Code: Select all

pip install -U nevergrad

Thanks again!

[chrisw]

You may try Lakas using nevergrad framework to optimize you engine's search and evaluation parameters.

Sample command line from optimization comparison. Help is here. Optimizers that are supported so far.

Hi Ferdy,

This is cool. Tried it out today on a few parameters and got a quick +30 Elo.
Setting depth=8, 9 and so on is fine, but I didn’t yet find a way to set time per move.
Is there a movetime=N option as well as the depth=d option?
Tried base-time set to 5 (secs) and depth=1000 but no help.

[Ferdy]

Hi Ferdy,

This is cool. Tried it out today on a few parameters and got a quick +30 Elo.
Setting depth=8, 9 and so on is fine, but I didn’t yet find a way to set time per move.
Is there a movetime=N option as well as the depth=d option?
Tried base-time set to 5 (secs) and depth=1000 but no help.

It does not support movetime yet. I will add it this weekend.

[chrisw]

Hi Ferdy,

This is cool. Tried it out today on a few parameters and got a quick +30 Elo.
Setting depth=8, 9 and so on is fine, but I didn’t yet find a way to set time per move.
Is there a movetime=N option as well as the depth=d option?
Tried base-time set to 5 (secs) and depth=1000 but no help.

It does not support movetime yet. I will add it this weekend.

Thanks. Depth+N paradigm works fine for evaluation features but not really for search (for example changes that reduce pruning and increase extensions will be favoured in a match environment where the engine is not time restricted).

One other very minor detail. The batch file that runs the SF demo is missing “—opening-file-format Epd”