Page 1 of 2
tuning info
Posted: Wed Jan 03, 2018 11:20 am
by elcabesa
Hi everyone,
I just finished a good session of tuning of my engine using a technique very similar to texel method, minimising the error of evaluation against game results.
I'd like now to try to tune some other parameters using an spsa like approach.
I have studied the perl spsa tuner and looked at fishtest code.
The spsa Perl tuner doesn't work very well with vajolet and I don't like to debug the uci interaction between them. We already have a very good tournament software, cutechess and I was asking myself if already exist some tool or fork of cutechess that allow to tune an engine.
Are you aware of some project or tuning script already using cutechess? Otherwise my idea is to add a tuner to cutechess.
Thank you all
Re: tuning info
Posted: Wed Jan 03, 2018 4:32 pm
by stegemma
elcabesa wrote:Hi everyone,
I just finished a good session of tuning of my engine using a technique very similar to texel method, minimising the error of evaluation against game results.
I'd like now to try to tune some other parameters using an spsa like approach.
I have studied the perl spsa tuner and looked at fishtest code.
The spsa Perl tuner doesn't work very well with vajolet and I don't like to debug the uci interaction between them. We already have a very good tournament software, cutechess and I was asking myself if already exist some tool or fork of cutechess that allow to tune an engine.
Are you aware of some project or tuning script already using cutechess? Otherwise my idea is to add a tuner to cutechess.
Thank you all
Does Vajolet have a parameter on the commadn line to load parameters from a file? I could test genetic optimization using Sabrina tournaments mode. I don't know SPSA but it could be added as a learning algorithm.
Re: tuning info
Posted: Wed Jan 03, 2018 4:53 pm
by Daniel Shawul
Clop + cutechess-cli is good for tuning search parameters.
It could be slow if you have too many parameters but often times there are just a handful of search parameters.
For example, I only have one futility_pruning_margin to tune in search.
Daniel
Re: tuning info
Posted: Thu Jan 04, 2018 7:05 pm
by elcabesa
I don't like clop, but probably I can his approach.
write a python script for spsa that use cutechess to play game.. I'll give it a try
Re: tuning info
Posted: Thu Jan 04, 2018 11:07 pm
by jdart
There is a script that comes with CLOP that you can probably modify.
You might also take a look also at:
https://github.com/jdart1/arasan-chess/ ... sitions.py
Which shows how to invoke cutechess-cli and parse the results.
--Jon
Re: tuning info
Posted: Fri Jan 05, 2018 1:04 am
by Daniel Shawul
I think clop may support different optimization algorithms.
I see an C2SPSA.cpp in there though i am not sure what it is used for.
P.S: Does anybody know why latest cutechess-cli does not display results on stdout ?
It plays the game and saves it to a pgn but there is no result output on the screen.
I tried cutechess-cli 1.0 and 0.85, on ubuntu 17.10 -- Qt version 5.9.1
Daniel
Re: tuning info
Posted: Fri Jan 05, 2018 4:04 pm
by petero2
Daniel Shawul wrote:P.S: Does anybody know why latest cutechess-cli does not display results on stdout ?
It plays the game and saves it to a pgn but there is no result output on the screen.
I tried cutechess-cli 1.0 and 0.85, on ubuntu 17.10 -- Qt version 5.9.1
I had this problem in fedora 24. I solved it by adding the following to ~/.config/QtProject/qtlogging.ini:
Re: tuning info
Posted: Fri Jan 05, 2018 6:26 pm
by Daniel Shawul
It works like a charm!
Thanks a lot Peter!
Daniel
Re: tuning info
Posted: Sun Jan 07, 2018 3:27 pm
by Ferdy
elcabesa wrote:The spsa Perl tuner doesn't work very well with vajolet and I don't like to debug the uci interaction between them.
Couple of notes when using spsa from
https://github.com/zamar/spsa/blob/master/spsa.pl
1. Be aware of the game adjudication on stalemate.
When your engine received a stalemate position what bestmove it prints?
Stockfish would print
If your engine returns bestmove 0000,
then the code in line 446,
Code: Select all
$flag_stalemate = 1 if ($array[1] eq '(none)');
should be change to:
Code: Select all
$flag_stalemate = 1 if ($array[1] eq '0000');
2. When parsing engine output info line, make sure the splitted line converted into an array is properly read. At line 452,
Code: Select all
# Check for mate in one
if ($#array >= 9 && $array[0] eq 'info' && $array[1] eq 'depth' &&
$array[7] eq 'score' && $array[8] eq 'mate' && $array[9] eq '1')
{
$flag_mate = 1;
$winner = $engine_to_move;
}
With typical uci engine search output for example.
Code: Select all
info depth 1 score mate 1 nodes 20 nps 1818 tbhits 0 time 11 pv e2e4
The code above must be revised to:
Code: Select all
if ($#array >= 5 && $array[0] eq 'info' && $array[1] eq 'depth' &&
$array[3] eq 'score' && $array[4] eq 'mate' && $array[5] eq '1')
{
...
}
3. There is an unsafe code in line 461, revised this depending on your engine output.
Code: Select all
# Record score
if ($#array >= 7 && $array[0] eq 'info' && $array[1] eq 'depth' &&
$array[7] eq 'score')
{
$score = $array[9] if ($array[8] eq 'cp');
$score = +100000 if ($array[8] eq 'mate' && $array[9] > 0);
$score = -100000 if ($array[8] eq 'mate' && $array[9] < 0);
}
That,
should be change to:
Re: tuning info
Posted: Sun Jan 07, 2018 6:05 pm
by elcabesa
I don't like reinventing the well, we already have the beatiful cutechess-cli to play games.
I think I'll probably port spsa script to python and use cutechess to play games