Laskas parameter optimizer

[Ferdy]

A few suggestions :
- try to not call "python" directly inside the script (not working when python interpreter is named python for instance)

I cannot understand what you mean by that, could you explain more.

- make cutechess executable a parameter

I plan to do it but not a priority.

- specifying depth and a time might confuse some engine (but I agree UCI protocol is respected)

I will modify it so that time or depth or both can be used.

- I think, for python3 at least, the popen shall look like this

Code: Select all

     process = Popen((str(tour_manager) + command).split(' '), stdout=PIPE, text=True)

My intention is for python 3. Does the current implementation does not work for you?

I'm trying it now, will report if it gives some Elo.

Thanks for the suggestions and testing.

[xr_a_y]

About python interpreter inside the code, I was refereing to the line

Code: Select all

tour_manager = 'python3 -u ./tourney_manager/duel/duel.py'

I tried an optimization over night using this parameters :

Code: Select all

python3 -u lakas.py \
        --optimizer oneplusone \
        --output-data-file oneplusone_t1.dat \
        --optimizer-log-file opt_log_t1.txt \
        --base-time-sec 3 \
	--inc-time-sec 0.03 \
        --budget 100 \
        --games-per-budget 1000 \
        --concurrency 7 \
        --engine /ssd/Minic/Tourney/minic_dev_linux_x64 \
        --input-param "{'nullMoveMinDepth': {'init':2, 'lower':0, 'upper':10}, 'nullMoveDynamicDivisor': {'init':180, 'lower':20, 'upper':600}}" \
        --opening-file /ssd/Minic/Book_and_Test/OpeningBook/Hert500.pgn \
        --opening-file-format pgn \
	--match-manager cutechess

It ran for 45 "budgets".

It first explored init param and then tried some couple of values : (66,3), (165,1), (317,0), (429,10)

But after that the optimizer is stuck on trying always (317,0) again and again, almost never trying something else aside from a (20,9), twice a (20,10) and a (600,10).

Is that expected ?

[Ferdy]

About python interpreter inside the code, I was refereing to the line

Code: Select all

tour_manager = 'python3 -u ./tourney_manager/duel/duel.py'

Right, I will try to make an improvement around it.

I tried an optimization over night using this parameters :

Code: Select all

python3 -u lakas.py \
        --optimizer oneplusone \
        --output-data-file oneplusone_t1.dat \
        --optimizer-log-file opt_log_t1.txt \
        --base-time-sec 3 \
	--inc-time-sec 0.03 \
        --budget 100 \
        --games-per-budget 1000 \
        --concurrency 7 \
        --engine /ssd/Minic/Tourney/minic_dev_linux_x64 \
        --input-param "{'nullMoveMinDepth': {'init':2, 'lower':0, 'upper':10}, 'nullMoveDynamicDivisor': {'init':180, 'lower':20, 'upper':600}}" \
        --opening-file /ssd/Minic/Book_and_Test/OpeningBook/Hert500.pgn \
        --opening-file-format pgn \
	--match-manager cutechess

It ran for 45 "budgets".

It first explored init param and then tried some couple of values : (66,3), (165,1), (317,0), (429,10)

But after that the optimizer is stuck on trying always (317,0) again and again, almost never trying something else aside from a (20,9), twice a (20,10) and a (600,10).

Is that expected ?

Yes, the optimizer oneplusone is an optimizer that repeats the parameter values as a way of verification while tuning progresses. Once it sees a parameter values that performs better it tries it more. There is control on how often it repeats.

From help:

Code: Select all

  --oneplusone-noise-handling ONEPLUSONE_NOISE_HANDLING
                        Parameter for oneplusone optimizer, can be optimistic or random,
                        or a tuple, default=optimistic.
                        Example:
                        --oneplusone-noise-handling random ...
                        --oneplusone-noise-handling optimistic ...
                        --oneplusone-noise-handling "('optimistic', 0.01)" ...
                        where:
                          0.01 is the coefficient (the regularity of reevaluations),
                          default coefficient is 0.05.

Example of controlling reevaluation.

Code: Select all

--oneplusone-noise-handling "('optimistic', 0.01)"

This optimizer is well suited for low number of games per budget, as it will reevaluate parameters with good history. Your games per budget of 1000 is very good. With high number of games per budget like 500 and above, I will use an optimizer other than oneplusone.

[Patrice Duhamel]

Include these command line options on your command line.

Code: Select all

--input-data-file out1.dat --output-data-file out1.dat

As tuning progresses, it will save the tuning data to out1.dat.
At startup it will read out1.dat and the optimizer would adjust its algo based on history on the out1.dat.

Thanks, I will try this.

[Patrice Duhamel]

Include these command line options on your command line.

Code: Select all

--input-data-file out1.dat --output-data-file out1.dat

As tuning progresses, it will save the tuning data to out1.dat.
At startup it will read out1.dat and the optimizer would adjust its algo based on history on the out1.dat.

I think it doesn't work, when I restart Lakas using the --input-data-file option, it doesn't remember the last best parameters, do I have to set manually the default parameters to the previous best parameters ?

[Ferdy]

Include these command line options on your command line.

Code: Select all

--input-data-file out1.dat --output-data-file out1.dat

As tuning progresses, it will save the tuning data to out1.dat.
At startup it will read out1.dat and the optimizer would adjust its algo based on history on the out1.dat.

I think it doesn't work, when I restart Lakas using the --input-data-file option, it doesn't remember the last best parameters,

When you resume you should see something like this:

Code: Select all

2021-01-27 21:50:45,774 | INFO  | oneplusone previous num_ask: 62

There is previous num_ask value. See your log_lakas.txt file.

do I have to set manually the default parameters to the previous best parameters ?

No.

[Ferdy]

To admin/moderators could you please change the title

from

Laskas parameter optimizer

to

Lakas parameter optimizer

This was my mistake sorry.

Thank you.

[Patrice Duhamel]

When you resume you should see something like this:

Code: Select all

2021-01-27 21:50:45,774 | INFO  | oneplusone previous num_ask: 62

There is previous num_ask value. See your log_lakas.txt file.

Yes I see a previous num_ask value when I restart, but every times I restart it uses the default init parameters as best value.

Example :

1st run for 13 budgets : found best parameters at budget 3, best loss = 0.435
2nd run for 16 budgets : found best parameters at budget 8, best loss = 0.48
3rd start for 15 budgets : never found best parameters (best = init value)

[Ferdy]

When you resume you should see something like this:

Code: Select all

2021-01-27 21:50:45,774 | INFO  | oneplusone previous num_ask: 62

There is previous num_ask value. See your log_lakas.txt file.

Yes I see a previous num_ask value when I restart, but every times I restart it uses the default init parameters as best value.

Example :

1st run for 13 budgets : found best parameters at budget 3, best loss = 0.435
2nd run for 16 budgets : found best parameters at budget 8, best loss = 0.48
3rd start for 15 budgets : never found best parameters (best = init value)

On this,

3rd start for 15 budgets : never found best parameters (best = init value)

After finishing the scheduled 15 budgets (not interrupted manually or by accident) it should recommend a different param values other than init values.


BTW which optimizer do you use, I will test it.

[Ferdy]

When you resume you should see something like this:

Code: Select all

2021-01-27 21:50:45,774 | INFO  | oneplusone previous num_ask: 62

There is previous num_ask value. See your log_lakas.txt file.

Yes I see a previous num_ask value when I restart, but every times I restart it uses the default init parameters as best value.

Example :

1st run for 13 budgets : found best parameters at budget 3, best loss = 0.435
2nd run for 16 budgets : found best parameters at budget 8, best loss = 0.48
3rd start for 15 budgets : never found best parameters (best = init value)

Right I see what you mean now.

For initial run of optimizer the best param is the init param.

In every restart, I printed that the best param is the init param (which is a mistake, will fix it later). But actually the best param should be ask from the optimizer or I have to maintain a record for it and during resumption I will display it.

After the scheduled budgets are completed, I ask the optimizer of the best param and printed it, this is the best param based from optimizer.