Laskas parameter optimizer

[Ferdy]

You may try Lakas using nevergrad framework to optimize you engine's search and evaluation parameters.

Sample command line from optimization comparison. Help is here. Optimizers that are supported so far.

[Ferdy]

Run an optimization combining search and evaluation parameters with promising result.

[Joerg Oster]

Run an optimization combining search and evaluation parameters with promising result.

Will you give them a try at Fishtest?

[Ferdy]

Run an optimization combining search and evaluation parameters with promising result.

Will you give them a try at Fishtest?

What should I do, I am not familiar with Fishtest.

[Joerg Oster]

Run an optimization combining search and evaluation parameters with promising result.

Will you give them a try at Fishtest?

What should I do, I am not familiar with Fishtest.

Fork the official repo. https://github.com/official-stockfish/Stockfish
Create a new branch with your changes.
Create an account at Fishtest. https://tests.stockfishchess.org/tests
Submit your test.

This guide https://github.com/glinscott/fishtest/w ... first-test may be of help.

[Patrice Duhamel]

I tried Lakas with Cheese, to optimize the mobility for knight, bishop, rook and queen. (score = (mobility * scale) / 8 - translate)
I ran Lakas 4 times (for each pieces), 4 parameters by piece, and I used the same options as the example but for only 30 budgets and 100 games / budgets, at 10s+0.1s.

Result with tuned mobility : 2500 games at 10s+0.1s :

Code: Select all

Rank Name                      Elo    +    - games score oppo. draws 
   1 Cheese-30-dev               4    6    6  2500   51%    -4   43% 
   2 Cheese-30-dev-optimized    -4    6    6  2500   49%     4   43% 

Should I run Lakas with all 16 parameters at once ? and use more than 30 budgets ?
I think 100 games/budget are not enough, but it already takes lots of time (I can use only 4 threads).

[Ferdy]

I tried Lakas with Cheese, to optimize the mobility for knight, bishop, rook and queen. (score = (mobility * scale) / 8 - translate)
I ran Lakas 4 times (for each pieces), 4 parameters by piece, and I used the same options as the example but for only 30 budgets and 100 games / budgets, at 10s+0.1s.

Result with tuned mobility : 2500 games at 10s+0.1s :

Code: Select all

Rank Name                      Elo    +    - games score oppo. draws 
   1 Cheese-30-dev               4    6    6  2500   51%    -4   43% 
   2 Cheese-30-dev-optimized    -4    6    6  2500   49%     4   43% 

Should I run Lakas with all 16 parameters at once ? and use more than 30 budgets ?
I think 100 games/budget are not enough, but it already takes lots of time (I can use only 4 threads).

What do you mean by "I ran Lakas 4 times (for each pieces), 4 parameters by piece,"?

Should I run Lakas with all 16 parameters at once ? and use more than 30 budgets ?

Yes you can do that.

More games per budget is always better and more budgets are also better especially if number of parameters are high. These optimizations are really expensive.

[Patrice Duhamel]

What do you mean by "I ran Lakas 4 times (for each pieces), 4 parameters by piece,"?

I ran it to optimize Knight mobility, then Bishop mobility, rook, and queen. (4 parameters by piece)
Because with 16 parameters at once I guess it needs more budgets and it takes too much time.

It's better to use more parameters at the same time ?

I will try again later when I change my computer.

[Ferdy]

What do you mean by "I ran Lakas 4 times (for each pieces), 4 parameters by piece,"?

I ran it to optimize Knight mobility, then Bishop mobility, rook, and queen. (4 parameters by piece)
Because with 16 parameters at once I guess it needs more budgets and it takes too much time.

It's better to use more parameters at the same time ?

Yes.

It has interrupt and resume feature. If you are busy at one point, just exit the tuning. Then continue with same command line that you had later. Better to save the command line in a batch file.

Include these command line options on your command line.

Code: Select all

--input-data-file out1.dat --output-data-file out1.dat

As tuning progresses, it will save the tuning data to out1.dat.
At startup it will read out1.dat and the optimizer would adjust its algo based on history on the out1.dat.

[xr_a_y]

Hi, great scripts. Thanks.

A few suggestions :
- try to not call "python" directly inside the script (not working when python interpreter is named python for instance)
- make cutechess executable a parameter
- specifying depth and a time might confuse some engine (but I agree UCI protocol is respected)
- I think, for python3 at least, the popen shall look like this

Code: Select all

     process = Popen((str(tour_manager) + command).split(' '), stdout=PIPE, text=True)

I'm trying it now, will report if it gives some Elo.