Jorg, have you tried one of the other base estimators supported by skopt/tune? They might be faster. You can "RF", "ET" or "GBRT". I'll think of other ways to speed up for many games. Maybe something like running more than 2 games for every choice of parameters.
But even inverting a 2000x2000 matrix shouldn't take more than a second on most machines, so hardly a big factor if each game your playing takes 5-30 seconds.
New Tool for Tuning with Skopt
Moderators: hgm, Rebel, chrisw
-
- Posts: 939
- Joined: Fri Mar 10, 2006 4:29 pm
- Location: Germany
Re: New Tool for Tuning with Skopt
Hi Thomas,thomasahle wrote: ↑Sun Sep 01, 2019 3:12 am Jorg, have you tried one of the other base estimators supported by skopt/tune? They might be faster. You can "RF", "ET" or "GBRT". I'll think of other ways to speed up for many games. Maybe something like running more than 2 games for every choice of parameters.
But even inverting a 2000x2000 matrix shouldn't take more than a second on most machines, so hardly a big factor if each game your playing takes 5-30 seconds.
no, I didn't try a different base estimator.
I tried 'LCB' as acq_func instead of 'gp_hedge', but it was only a marginal speedup.
I also tried playing games without a book, but this also made no noticeable difference in terms of speed.
And just to clarify, whenever I stop the tuning script because the calculation between the games takes too long,
we're talking here about 10 - 30 (and growing) seconds!
It also seems like Jonathan didn't experience this kind of slowdown. Strange enough.
Jörg Oster
-
- Posts: 2272
- Joined: Mon Sep 29, 2008 1:50 am
Re: New Tool for Tuning with Skopt
Maybe you are running out of memory?Joerg Oster wrote: ↑Sun Sep 01, 2019 10:18 amHi Thomas,thomasahle wrote: ↑Sun Sep 01, 2019 3:12 am Jorg, have you tried one of the other base estimators supported by skopt/tune? They might be faster. You can "RF", "ET" or "GBRT". I'll think of other ways to speed up for many games. Maybe something like running more than 2 games for every choice of parameters.
But even inverting a 2000x2000 matrix shouldn't take more than a second on most machines, so hardly a big factor if each game your playing takes 5-30 seconds.
no, I didn't try a different base estimator.
I tried 'LCB' as acq_func instead of 'gp_hedge', but it was only a marginal speedup.
I also tried playing games without a book, but this also made no noticeable difference in terms of speed.
And just to clarify, whenever I stop the tuning script because the calculation between the games takes too long,
we're talking here about 10 - 30 (and growing) seconds!
It also seems like Jonathan didn't experience this kind of slowdown. Strange enough.
Ideas=science. Simplification=engineering.
Without ideas there is nothing to simplify.
Without ideas there is nothing to simplify.
-
- Posts: 939
- Joined: Fri Mar 10, 2006 4:29 pm
- Location: Germany
Re: New Tool for Tuning with Skopt
So I just tried 'GBRT' as base estimator (with 'LCB' as acq_func, although this seems to be the default anyway).Joerg Oster wrote: ↑Sun Sep 01, 2019 10:18 amHi Thomas,thomasahle wrote: ↑Sun Sep 01, 2019 3:12 am Jorg, have you tried one of the other base estimators supported by skopt/tune? They might be faster. You can "RF", "ET" or "GBRT". I'll think of other ways to speed up for many games. Maybe something like running more than 2 games for every choice of parameters.
But even inverting a 2000x2000 matrix shouldn't take more than a second on most machines, so hardly a big factor if each game your playing takes 5-30 seconds.
no, I didn't try a different base estimator.
I tried 'LCB' as acq_func instead of 'gp_hedge', but it was only a marginal speedup.
I also tried playing games without a book, but this also made no noticeable difference in terms of speed.
And just to clarify, whenever I stop the tuning script because the calculation between the games takes too long,
we're talking here about 10 - 30 (and growing) seconds!
It also seems like Jonathan didn't experience this kind of slowdown. Strange enough.
Good news, from begin to the end it was running smoothly and at the same speed!
Here is the final result:
Code: Select all
Summarizing best values
Best expectation (κ=0): [ 8 48 28 99 9 35 69 7 0 0] = -0.000 ± 0.824 (ELO-diff -0.000 ± 405.717)
Best expectation (κ=1): [34 18 46 81 28 36 8 80 3 37] = -0.000 ± 0.403 (ELO-diff -0.000 ± 148.399)
Best expectation (κ=2): [34 18 46 81 28 36 8 80 3 37] = -0.000 ± 0.403 (ELO-diff -0.000 ± 148.399)
Best expectation (κ=3): [34 18 46 81 28 36 8 80 3 37] = -0.000 ± 0.403 (ELO-diff -0.000 ± 148.399)
Jörg Oster
-
- Posts: 939
- Joined: Fri Mar 10, 2006 4:29 pm
- Location: Germany
Re: New Tool for Tuning with Skopt
How can I find out?Michel wrote: ↑Sun Sep 01, 2019 11:06 amMaybe you are running out of memory?Joerg Oster wrote: ↑Sun Sep 01, 2019 10:18 amHi Thomas,thomasahle wrote: ↑Sun Sep 01, 2019 3:12 am Jorg, have you tried one of the other base estimators supported by skopt/tune? They might be faster. You can "RF", "ET" or "GBRT". I'll think of other ways to speed up for many games. Maybe something like running more than 2 games for every choice of parameters.
But even inverting a 2000x2000 matrix shouldn't take more than a second on most machines, so hardly a big factor if each game your playing takes 5-30 seconds.
no, I didn't try a different base estimator.
I tried 'LCB' as acq_func instead of 'gp_hedge', but it was only a marginal speedup.
I also tried playing games without a book, but this also made no noticeable difference in terms of speed.
And just to clarify, whenever I stop the tuning script because the calculation between the games takes too long,
we're talking here about 10 - 30 (and growing) seconds!
It also seems like Jonathan didn't experience this kind of slowdown. Strange enough.
Total sytem memory is 8 GB.
Note, a testrun with 'GBRT' as base estimator just finished smoothly!
And from the API documentation here https://scikit-optimize.github.io/, I don't immediately see why the 'GP' method should take so much more computation time.
(But I am only a noob, of course!)
Jörg Oster
-
- Posts: 939
- Joined: Fri Mar 10, 2006 4:29 pm
- Location: Germany
Re: New Tool for Tuning with Skopt
'top' tells me the python3 process stays permanently below 5% of memory usage.Michel wrote: ↑Sun Sep 01, 2019 11:06 amMaybe you are running out of memory?Joerg Oster wrote: ↑Sun Sep 01, 2019 10:18 amHi Thomas,thomasahle wrote: ↑Sun Sep 01, 2019 3:12 am Jorg, have you tried one of the other base estimators supported by skopt/tune? They might be faster. You can "RF", "ET" or "GBRT". I'll think of other ways to speed up for many games. Maybe something like running more than 2 games for every choice of parameters.
But even inverting a 2000x2000 matrix shouldn't take more than a second on most machines, so hardly a big factor if each game your playing takes 5-30 seconds.
no, I didn't try a different base estimator.
I tried 'LCB' as acq_func instead of 'gp_hedge', but it was only a marginal speedup.
I also tried playing games without a book, but this also made no noticeable difference in terms of speed.
And just to clarify, whenever I stop the tuning script because the calculation between the games takes too long,
we're talking here about 10 - 30 (and growing) seconds!
It also seems like Jonathan didn't experience this kind of slowdown. Strange enough.
And at that time the process already started to show a noticeable slowdown.
Code: Select all
PID USER PR NI VIRT RES SHR S %CPU %MEM ZEIT+ BEFEHL
28328 joster 20 0 807180 367740 31776 R 150,5 4,5 8:29.91 python3
Jörg Oster
-
- Posts: 939
- Joined: Fri Mar 10, 2006 4:29 pm
- Location: Germany
Re: New Tool for Tuning with Skopt
Is this of any help?thomasahle wrote: ↑Sun Sep 01, 2019 3:12 am Jorg, have you tried one of the other base estimators supported by skopt/tune? They might be faster. You can "RF", "ET" or "GBRT". I'll think of other ways to speed up for many games. Maybe something like running more than 2 games for every choice of parameters.
But even inverting a 2000x2000 matrix shouldn't take more than a second on most machines, so hardly a big factor if each game your playing takes 5-30 seconds.
Code: Select all
Summarizing best values
unhandled exception during asyncio.run() shutdown
task: <Task finished coro=<main() done, defined at tune.py:299> exception=ValueError('math domain error')>
Traceback (most recent call last):
File "/usr/lib/python3.7/asyncio/runners.py", line 43, in run
return loop.run_until_complete(main)
File "/usr/lib/python3.7/asyncio/base_events.py", line 571, in run_until_complete
self.run_forever()
File "/usr/lib/python3.7/asyncio/base_events.py", line 539, in run_forever
self._run_once()
File "/usr/lib/python3.7/asyncio/base_events.py", line 1775, in _run_once
handle._run()
File "/usr/lib/python3.7/asyncio/events.py", line 88, in _run
self._context.run(self._callback, *self._args)
File "tune.py", line 271, in on_done
opt.tell(x, -y) # opt is minimizing
File "/home/joster/.local/lib/python3.7/site-packages/skopt/optimizer/optimizer.py", line 443, in tell
return self._tell(x, y, fit=fit)
File "/home/joster/.local/lib/python3.7/site-packages/skopt/optimizer/optimizer.py", line 502, in _tell
acq_func_kwargs=self.acq_func_kwargs)
File "/home/joster/.local/lib/python3.7/site-packages/skopt/acquisition.py", line 42, in _gaussian_acquisition
func_and_grad = gaussian_lcb(X, model, kappa, return_grad)
File "/home/joster/.local/lib/python3.7/site-packages/skopt/acquisition.py", line 142, in gaussian_lcb
mu, std = model.predict(X, return_std=True)
File "/home/joster/.local/lib/python3.7/site-packages/skopt/learning/gaussian_process/gpr.py", line 324, in predict
y_var -= np.einsum("ki,kj,ij->k", K_trans, K_trans, K_inv)
File "<__array_function__ internals>", line 6, in einsum
File "/home/joster/.local/lib/python3.7/site-packages/numpy/core/einsumfunc.py", line 1356, in einsum
return c_einsum(*operands, **kwargs)
KeyboardInterrupt
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "tune.py", line 414, in main
pm = max(abs(score_to_elo(y_pred[i] / 2 + sigma[i] / 2) - elo),
File "tune.py", line 412, in score_to_elo
return - 400 * math.log10(2 / (score + 1) - 1)
ValueError: math domain error
Jörg Oster
-
- Posts: 77
- Joined: Mon Apr 16, 2018 6:56 pm
Re: New Tool for Tuning with Skopt
I had the same error using RF as -base-estimator.
Other issue i had: When data file has 100 valid lines and the number of iterations (-n) option is 100, it will give me 'Not enought data to summarize results.'
Other issue i had: When data file has 100 valid lines and the number of iterations (-n) option is 100, it will give me 'Not enought data to summarize results.'
-
- Posts: 4833
- Joined: Sun Aug 10, 2008 3:15 pm
- Location: Philippines
Re: New Tool for Tuning with Skopt
Tried to monitor memory and cpu usage of tune.py. I tuned 6 eval parameters, concurrency is 6 and set at 1000 games and 50ms/move, others are default.
Started monitoring after 100 games, and after around 520/1000 games I stopped the tuner. Memory usage increases as games increases. CPU usage also increases but still within 2/8 threads or 25%.
This is on win7, 12GB, and 8 threads from 4-cores, from the plot tune.py has reach around 1.5GB.
mem_rate_increase = (1500-550)MB / (9500-0)s or around 100kB/s
I think with this setting my hardware can safely reach 600 games without problems.
Started monitoring after 100 games, and after around 520/1000 games I stopped the tuner. Memory usage increases as games increases. CPU usage also increases but still within 2/8 threads or 25%.
This is on win7, 12GB, and 8 threads from 4-cores, from the plot tune.py has reach around 1.5GB.
mem_rate_increase = (1500-550)MB / (9500-0)s or around 100kB/s
I think with this setting my hardware can safely reach 600 games without problems.
-
- Posts: 939
- Joined: Fri Mar 10, 2006 4:29 pm
- Location: Germany
Re: New Tool for Tuning with Skopt
I observe similar memory consumption wth base estimator set to 'RF'.Ferdy wrote: ↑Sun Sep 01, 2019 2:08 pm Tried to monitor memory and cpu usage of tune.py. I tuned 6 eval parameters, concurrency is 6 and set at 1000 games and 50ms/move, others are default.
Started monitoring after 100 games, and after around 520/1000 games I stopped the tuner. Memory usage increases as games increases. CPU usage also increases but still within 2/8 threads or 25%.
This is on win7, 12GB, and 8 threads from 4-cores, from the plot tune.py has reach around 1.5GB.
mem_rate_increase = (1500-550)MB / (9500-0)s or around 100kB/s
I think with this setting my hardware can safely reach 600 games without problems.
Code: Select all
PID USER PR NI VIRT RES SHR S %CPU %MEM ZEIT+ BEFEHL
1097 joster 20 0 3933736 3,363g 30580 R 99,7 43,5 92:34.85 python3
so I stopped here.
Jörg Oster