Hi Thomas,
I have a version of Myrddin ready for you to test if you want. PM me your e-mail address and I'll send it off.
Thanks for your help!
jm
New Tool for Tuning with Skopt
Moderators: hgm, Rebel, chrisw
-
- Posts: 1357
- Joined: Wed Mar 08, 2006 10:15 pm
- Location: San Francisco, California
-
- Posts: 1056
- Joined: Fri Mar 10, 2006 6:07 am
- Location: Basque Country (Spain)
Re: New Tool for Tuning with Skopt
I find the new version that gives more comments in pgn file as interesting pgn.
But I see that PV is also shown on the console and this is not good for me, I try to play 7 concurrent games and litte movetime, so seeing all the PV seems more annoying than interesting information, besides not being able to see the boards . In addition, I believe that if PV information is printed in console, this possibly slows the process.
But I see that PV is also shown on the console and this is not good for me, I try to play 7 concurrent games and litte movetime, so seeing all the PV seems more annoying than interesting information, besides not being able to see the boards . In addition, I believe that if PV information is printed in console, this possibly slows the process.
-
- Posts: 94
- Joined: Thu Feb 27, 2014 8:19 pm
-
- Posts: 360
- Joined: Thu Jan 22, 2015 3:21 pm
- Location: Zurich, Switzerland
- Full name: Jonathan Rosenthal
Re: New Tool for Tuning with Skopt
Just wanted to mention I was successfully able to retune Winter's futility and static null move pruning margins with this tool. I measured a self play rating gain of +9 +- 3 Elo roughly.
-Jonathan
-
- Posts: 77
- Joined: Mon Apr 16, 2018 6:56 pm
Re: New Tool for Tuning with Skopt
Hello,
It's a nice project, but i had issues:
- ahash from log file should not consider number of games and even concurrency, that way would be easy to run a 200 game session run using 100 games from previous run.
- Let me use options that are not seen as UCI options, i don't wanna to expose all possible tunable constants as options. I just need to define them using tool input options.
- Can't make it work with more than 100 games, tested with 6 options.
It's a nice project, but i had issues:
- ahash from log file should not consider number of games and even concurrency, that way would be easy to run a 200 game session run using 100 games from previous run.
- Let me use options that are not seen as UCI options, i don't wanna to expose all possible tunable constants as options. I just need to define them using tool input options.
- Can't make it work with more than 100 games, tested with 6 options.
-
- Posts: 937
- Joined: Fri Mar 10, 2006 4:29 pm
- Location: Germany
Re: New Tool for Tuning with Skopt
What happens with more than 100 games?Ratosh wrote: ↑Fri Aug 30, 2019 9:09 am Hello,
It's a nice project, but i had issues:
- ahash from log file should not consider number of games and even concurrency, that way would be easy to run a 200 game session run using 100 games from previous run.
- Let me use options that are not seen as UCI options, i don't wanna to expose all possible tunable constants as options. I just need to define them using tool input options.
- Can't make it work with more than 100 games, tested with 6 options.
I do have problems, too, when the number of games keeps rising.
CPU usage is suddenly growing with peaks up to >90% under Linux.
Jörg Oster
-
- Posts: 1056
- Joined: Fri Mar 10, 2006 6:07 am
- Location: Basque Country (Spain)
Re: New Tool for Tuning with Skopt
Can you indicate what parameters you used when calling tune.py?
I was also tuning those 2 parameters
-
- Posts: 937
- Joined: Fri Mar 10, 2006 4:29 pm
- Location: Germany
Re: New Tool for Tuning with Skopt
It looks like the computational effort is steadily increasing with every iteration, especially when tuning several parameters at once.Joerg Oster wrote: ↑Fri Aug 30, 2019 10:35 amWhat happens with more than 100 games?Ratosh wrote: ↑Fri Aug 30, 2019 9:09 am Hello,
It's a nice project, but i had issues:
- ahash from log file should not consider number of games and even concurrency, that way would be easy to run a 200 game session run using 100 games from previous run.
- Let me use options that are not seen as UCI options, i don't wanna to expose all possible tunable constants as options. I just need to define them using tool input options.
- Can't make it work with more than 100 games, tested with 6 options.
I do have problems, too, when the number of games keeps rising.
CPU usage is suddenly growing with peaks up to >90% under Linux.
I had to cancel another testrun planned with 2,000 iterations after only about 400 iterations, unfortunately. Further progress had become exceptionally expensive.
This is really bad news because this makes this tool almost unusable in practice. But maybe this can be fixed.
Good news is that the parameters were heading towards reasonable values.
(I was tuning some pawn values of Stockfish to validate the tuner is working ...)
Jörg Oster
-
- Posts: 2272
- Joined: Mon Sep 29, 2008 1:50 am
Re: New Tool for Tuning with Skopt
I think that Bayesian optimization is computationally very heavy. From the description here https://arxiv.org/pdf/1807.02811.pdf it appears to involve at every step the inversion of a matrix whose size is the number of observations up to then.Joerg Oster wrote: ↑Fri Aug 30, 2019 6:33 pmIt looks like the computational effort is steadily increasing with every iteration, especially when tuning several parameters at once.Joerg Oster wrote: ↑Fri Aug 30, 2019 10:35 amWhat happens with more than 100 games?Ratosh wrote: ↑Fri Aug 30, 2019 9:09 am Hello,
It's a nice project, but i had issues:
- ahash from log file should not consider number of games and even concurrency, that way would be easy to run a 200 game session run using 100 games from previous run.
- Let me use options that are not seen as UCI options, i don't wanna to expose all possible tunable constants as options. I just need to define them using tool input options.
- Can't make it work with more than 100 games, tested with 6 options.
I do have problems, too, when the number of games keeps rising.
CPU usage is suddenly growing with peaks up to >90% under Linux.
I had to cancel another testrun planned with 2,000 iterations after only about 400 iterations, unfortunately. Further progress had become exceptionally expensive.
This is really bad news because this makes this tool almost unusable in practice. But maybe this can be fixed.
Good news is that the parameters were heading towards reasonable values.
(I was tuning some pawn values of Stockfish to validate the tuner is working ...)
Ideas=science. Simplification=engineering.
Without ideas there is nothing to simplify.
Without ideas there is nothing to simplify.
-
- Posts: 937
- Joined: Fri Mar 10, 2006 4:29 pm
- Location: Germany
Re: New Tool for Tuning with Skopt
So it's eventually better suited for tuning search parameters where you don't want to play ultrafast games.Michel wrote: ↑Sat Aug 31, 2019 3:21 pmI think that Bayesian optimization is computationally very heavy. From the description here https://arxiv.org/pdf/1807.02811.pdf it appears to involve at every step the inversion of a matrix whose size is the number of observations up to then.Joerg Oster wrote: ↑Fri Aug 30, 2019 6:33 pmIt looks like the computational effort is steadily increasing with every iteration, especially when tuning several parameters at once.Joerg Oster wrote: ↑Fri Aug 30, 2019 10:35 amWhat happens with more than 100 games?Ratosh wrote: ↑Fri Aug 30, 2019 9:09 am Hello,
It's a nice project, but i had issues:
- ahash from log file should not consider number of games and even concurrency, that way would be easy to run a 200 game session run using 100 games from previous run.
- Let me use options that are not seen as UCI options, i don't wanna to expose all possible tunable constants as options. I just need to define them using tool input options.
- Can't make it work with more than 100 games, tested with 6 options.
I do have problems, too, when the number of games keeps rising.
CPU usage is suddenly growing with peaks up to >90% under Linux.
I had to cancel another testrun planned with 2,000 iterations after only about 400 iterations, unfortunately. Further progress had become exceptionally expensive.
This is really bad news because this makes this tool almost unusable in practice. But maybe this can be fixed.
Good news is that the parameters were heading towards reasonable values.
(I was tuning some pawn values of Stockfish to validate the tuner is working ...)
Maybe I will give this another try.
Jörg Oster