Lc0 is the cuDNN version with default settings, I am tired of fiddling with its parameters, with performances varying at time controls, and it seems anyway there are some serious bugs.
So I finished the CLOP run after 1500 games, which took a full 4 days to run. Details are: 1m+1s, used id338 (after healing so evals are not crazy), 3 opponents to avoid bias in favor of one (rated around 3080-3090 CCRL each), randomized openings suite. Computer tested on has GTX1060 6GB.
Final result is cPUCT=3.15 and FPU=0.17
"Tactics are the bricks and sticks that make up a game, but positional play is the architectural blueprint."
Lc0 is the cuDNN version with default settings, I am tired of fiddling with its parameters, with performances varying at time controls, and it seems anyway there are some serious bugs.
So I finished the CLOP run after 1500 games, which took a full 4 days to run. Details are: 1m+1s, used id338 (after healing so evals are not crazy), 3 opponents to avoid bias in favor of one (rated around 3080-3090 CCRL each), randomized openings suite. Computer tested on has GTX1060 6GB.
Final result is cPUCT=3.15 and FPU=0.17
Thanks Albert for your great, lengthy work. I will put your settings compared to default against Arasan 20.5 in a gauntlet of 1600 ultra-fast games total, to see if the settings help. One of the problems of this LCzero in general is that it is scaling in some uncharted ways, and the settings may vary with time control, but if I will get something with your settings in ultra-fast games, then some confidence of fast testing may open new ways to optimize it. I will post the results in some maybe 7 hours, it still takes some time.
Laskos wrote: ↑Wed May 30, 2018 1:23 pm
Thanks Albert for your great, lengthy work. I will put your settings compared to default against Arasan 20.5 in a gauntlet of 1600 ultra-fast games total, to see if the settings help. One of the problems of this LCzero in general is that it is scaling in some uncharted ways, and the settings may vary with time control, but if I will get something with your settings in ultra-fast games, then some confidence of fast testing may open new ways to optimize it. I will post the results in some maybe 7 hours, it still takes some time.
I don't believe changing PUCT from 3 to 3.15 or FPUR from -0.05 to 0.15 would have impact anywhere near that you can actually measure it reliably with your sample of games. The idea that you can actually optimize a relatively complex thing such as UCT working in conjunction with complex two-head ResNet with just 2 fixed numbers is by itself pretty ridiculous. But anyway good luck in burning electricity.
Laskos wrote: ↑Wed May 30, 2018 1:56 am
From NN342 to NN352 it indeed seems to be stalling against an A/B engine Arasan 20.5. Here is the results of gauntlet:
Lc0 is the cuDNN version with default settings, I am tired of fiddling with its parameters, with performances varying at time controls, and it seems anyway there are some serious bugs.
So I finished the CLOP run after 1500 games, which took a full 4 days to run. Details are: 1m+1s, used id338 (after healing so evals are not crazy), 3 opponents to avoid bias in favor of one (rated around 3080-3090 CCRL each), randomized openings suite. Computer tested on has GTX1060 6GB.
Final result is cPUCT=3.15 and FPU=0.17
Thanks Albert for your great, lengthy work. I will put your settings compared to default against Arasan 20.5 in a gauntlet of 1600 ultra-fast games total, to see if the settings help. One of the problems of this LCzero in general is that it is scaling in some uncharted ways, and the settings may vary with time control, but if I will get something with your settings in ultra-fast games, then some confidence of fast testing may open new ways to optimize it. I will post the results in some maybe 7 hours, it still takes some time.
I m testing these settings with Lc0 Cudnn id 357 againt Houdini 1.5 at 1 min + 1 sec. Lc0 is much better from my point of view in attack now. He sees wins many move before Houdini does and seems to not mess these winning position like he did.
Laskos wrote: ↑Wed May 30, 2018 1:23 pm
Thanks Albert for your great, lengthy work. I will put your settings compared to default against Arasan 20.5 in a gauntlet of 1600 ultra-fast games total, to see if the settings help. One of the problems of this LCzero in general is that it is scaling in some uncharted ways, and the settings may vary with time control, but if I will get something with your settings in ultra-fast games, then some confidence of fast testing may open new ways to optimize it. I will post the results in some maybe 7 hours, it still takes some time.
I don't believe changing PUCT from 3 to 3.15 or FPUR from -0.05 to 0.15 would have impact anywhere near that you can actually measure it reliably with your sample of games. The idea that you can actually optimize a relatively complex thing such as UCT working in conjunction with complex two-head ResNet with just 2 fixed numbers is by itself pretty ridiculous. But anyway good luck in burning electricity.
No, the defaults are PUCT 1.2 and FPUR 0.20. Let's see, although I don't expect a large improvement. But earlier CLOP results were marred by many issues.
Werewolf wrote: ↑Mon May 28, 2018 10:25 pm
To me it looks like stalling. I wonder if this can be resurrected.
How can you "resurrect" something that's not dead?
Well.....323 to the present looks a lot like stalling to me. Even the inflated self-play elo isn't going up. Maybe it's a temporary stall...maybe it's not.
Laskos wrote: ↑Wed May 30, 2018 1:23 pm
Thanks Albert for your great, lengthy work. I will put your settings compared to default against Arasan 20.5 in a gauntlet of 1600 ultra-fast games total, to see if the settings help. One of the problems of this LCzero in general is that it is scaling in some uncharted ways, and the settings may vary with time control, but if I will get something with your settings in ultra-fast games, then some confidence of fast testing may open new ways to optimize it. I will post the results in some maybe 7 hours, it still takes some time.
I played 1500 games, probably I typed 1500 instead of 1600, but it doesn't matter, the result seems conclusive:
So, the new CLOP settings are 43 +/- 32 Elo points weaker than the default settings at 0.1 s/move. Probably settings depend on time control (and hardware, but we have identical GTX 1060 6GB), as I myself observed. It's a pain playing with them.
Laskos wrote: ↑Wed May 30, 2018 1:23 pm
Thanks Albert for your great, lengthy work. I will put your settings compared to default against Arasan 20.5 in a gauntlet of 1600 ultra-fast games total, to see if the settings help. One of the problems of this LCzero in general is that it is scaling in some uncharted ways, and the settings may vary with time control, but if I will get something with your settings in ultra-fast games, then some confidence of fast testing may open new ways to optimize it. I will post the results in some maybe 7 hours, it still takes some time.
I played 1500 games, probably I typed 1500 instead of 1600, but it doesn't matter, the result seems conclusive:
So, the new CLOP settings are 43 +/- 32 Elo points weaker than the default settings at 0.1 s/move. Probably settings depend on time control (and hardware, but we have identical GTX 1060 6GB), as I myself observed. It's a pain playing with them.
And if you ran it at the same TC as Adam, they'd be better then default, and then again if you ran it at longer TC they might again become worse.
You can't model different depth with 2 fixed parameters. It is hopeless and I really don't understand how ppl don't see such a simple thing. Imagine A/B search instead of null move, LMR, all kind of depth dependent tricks have only 2 fixed numbers that control shape of the search tree. How well you think it would work?
Laskos wrote: ↑Wed May 30, 2018 1:23 pm
Thanks Albert for your great, lengthy work. I will put your settings compared to default against Arasan 20.5 in a gauntlet of 1600 ultra-fast games total, to see if the settings help. One of the problems of this LCzero in general is that it is scaling in some uncharted ways, and the settings may vary with time control, but if I will get something with your settings in ultra-fast games, then some confidence of fast testing may open new ways to optimize it. I will post the results in some maybe 7 hours, it still takes some time.
I played 1500 games, probably I typed 1500 instead of 1600, but it doesn't matter, the result seems conclusive:
So, the new CLOP settings are 43 +/- 32 Elo points weaker than the default settings at 0.1 s/move. Probably settings depend on time control (and hardware, but we have identical GTX 1060 6GB), as I myself observed. It's a pain playing with them.
You're aware that the engine already has a 100ms (0.1s) overhead in the UCI settings, so basically you are running instant moves. I don't doubt your results, but I think you must be aware already that Leela does not scale in any way like normal A/B engines. My testing was already at an average 1-2 seconds per move. For me an engine is for analyzing and playing, and I don't know anyone who does either at a pace of microseconds thus I so see no point tuning for them.
"Tactics are the bricks and sticks that make up a game, but positional play is the architectural blueprint."
At the moment, I play two gauntlets on my third notebook: Komodo 5 vs. Leela Cuda with default settings and Komodo 5 vs. Leela Cuda with the older Clop-Settings (cpuct=3.168, fpu reduction=-0.0683). With 5'+3“ thinking time. I want to play at least 200 games, each. When this is done, I will try your new Clop-tuned values, too. It is interesting, that both Clop-tunings gave nearly the same cpuct-value (around 3.1). Only the fpu-reduction is very different.