Hai wrote: ↑Wed Jan 22, 2020 5:13 pm
Do we have now new LC0 default settings?
The tuned settings were done using a mix of short time controls using the T58 (58613) network. These settings should be approximately optimal for all time controls with <20k nodes / move. Even though the CPuct scaling curve is not uniquely determined on slower time controls, it can perform reasonably well on time controls up to <100k nodes, but is likely slightly suboptimal there (though it seemed to perform quite well for T60 on CCC's time control). For 1M+ nodes: no guarantees
I am currently in the process of writing a distributed tuning framework for lc0, which we then could utilize to tune slower time controls than before. These parameters will then likely be better candidates as new defaults, than the currently circulating ones.
Score of SV_256x20_1541_Kiudee vs SV_256x20_1541: 52 - 30 - 118 [0.555] 200
Elo difference: 38.37 +/- 30.78
Finished match
On Twitch (potato_chess), 2 testruns with Net 62038 were played (1'+2'' on RTX, which is not that short and 62038 is a bigger Net (24x320)). 200 games each. And with Kiudee, the result was +36 Elo better, than the default setting. So, in each testrun, I have seen, Kiudee scored at least +30 Elo better. Nice!
I will repeat my longtime-testrun (8'+5'') of Lc0 0.23.1 LS 12.2 with Kiudee-setting. Lets see, if there is an Elo-gain with long thinkingtime, too...but this will take 6 days from now.
Good! 1 + 2'' is already Blitz (large increment), so the optimum CPuct value is probably only smoothly increasing to LTC. Thanks for your test at 8' + 5'' in as many as 300 games, a huge task for a single machine. But it is needed, as these Kiudee parameters give a hard to believe boost. In the past I tried to fiddle with Lc0 engine parameters, it was always within even 1 SD error margins, it is hard to have many games at acceptable TC with Leela. Besides that, I do not understand what many of these parameters do precisely, and just blindly trying here and there is useless.
Thanks for posting them, now I will modify Cutechess json file for Lc0 0232 with different nets, all to take these parameters.
The repeated NN-longtime testrun of Lc0 0.23.1 LS12.2 with Kiudee-setting is finished. Impressive progress with long thinking-time, too.
Score of SV_256x20_1541_Kiudee vs SV_256x20_1541: 52 - 30 - 118 [0.555] 200
Elo difference: 38.37 +/- 30.78
Finished match
On Twitch (potato_chess), 2 testruns with Net 62038 were played (1'+2'' on RTX, which is not that short and 62038 is a bigger Net (24x320)). 200 games each. And with Kiudee, the result was +36 Elo better, than the default setting. So, in each testrun, I have seen, Kiudee scored at least +30 Elo better. Nice!
I will repeat my longtime-testrun (8'+5'') of Lc0 0.23.1 LS 12.2 with Kiudee-setting. Lets see, if there is an Elo-gain with long thinkingtime, too...but this will take 6 days from now.
Good! 1 + 2'' is already Blitz (large increment), so the optimum CPuct value is probably only smoothly increasing to LTC. Thanks for your test at 8' + 5'' in as many as 300 games, a huge task for a single machine. But it is needed, as these Kiudee parameters give a hard to believe boost. In the past I tried to fiddle with Lc0 engine parameters, it was always within even 1 SD error margins, it is hard to have many games at acceptable TC with Leela. Besides that, I do not understand what many of these parameters do precisely, and just blindly trying here and there is useless.
Thanks for posting them, now I will modify Cutechess json file for Lc0 0232 with different nets, all to take these parameters.
The repeated NN-longtime testrun of Lc0 0.23.1 LS12.2 with Kiudee-setting is finished. Impressive progress with long thinking-time, too.
Excellent, thank you very much! That was a very long run to test at 8 + 5 in 300 games. It confirms my results with T59 128x10 net, which is about 5 times faster in NPS at a similar number of nodes. I will maybe open a separate thread on my results using Kiudee setting settings with the small and fast T59 net. T59 and T60 are trained with very similar parameters as noise etc., but T40 and LS were trained a bit differently, so our results shows that Kiudee settings work for T40 and other nets too to LTC.
My results with T60, T59, T40 nets show about some 40 Elo points improvement at about 50k nodes per move against a variety of strong opponents, and as yours, 30 Elo points at significantly longer TC (200-500k nodes per move range). Now I am testing Kiudee settings with one modification for LTC, CPuct=2.600 instead of 2.147 for LTC. The first results are encouraging, but the test (500 games with T59 net) will be finished in 2 more days. Anyway, I think most with a RTX GPU should switch to Kiudee settings instead of the defaults, the improvement is significant across the ranges of TC and nets.
Score of SV_256x20_1541_Kiudee vs SV_256x20_1541: 52 - 30 - 118 [0.555] 200
Elo difference: 38.37 +/- 30.78
Finished match
On Twitch (potato_chess), 2 testruns with Net 62038 were played (1'+2'' on RTX, which is not that short and 62038 is a bigger Net (24x320)). 200 games each. And with Kiudee, the result was +36 Elo better, than the default setting. So, in each testrun, I have seen, Kiudee scored at least +30 Elo better. Nice!
I will repeat my longtime-testrun (8'+5'') of Lc0 0.23.1 LS 12.2 with Kiudee-setting. Lets see, if there is an Elo-gain with long thinkingtime, too...but this will take 6 days from now.
Good! 1 + 2'' is already Blitz (large increment), so the optimum CPuct value is probably only smoothly increasing to LTC. Thanks for your test at 8' + 5'' in as many as 300 games, a huge task for a single machine. But it is needed, as these Kiudee parameters give a hard to believe boost. In the past I tried to fiddle with Lc0 engine parameters, it was always within even 1 SD error margins, it is hard to have many games at acceptable TC with Leela. Besides that, I do not understand what many of these parameters do precisely, and just blindly trying here and there is useless.
Thanks for posting them, now I will modify Cutechess json file for Lc0 0232 with different nets, all to take these parameters.
The repeated NN-longtime testrun of Lc0 0.23.1 LS12.2 with Kiudee-setting is finished. Impressive progress with long thinking-time, too.
I agree. I think the key point is if you are testing with default. You need to understand default in not close to the better settings with the NN. And better settings keep being found.
"The worst thing that can happen to a forum is a running wild attacking moderator(HGM) who is not corrected by the community." - Ed Schröder
But my words like silent raindrops fell. And echoed in the wells of silence.
pohl4711 wrote: ↑Sun Jan 26, 2020 5:13 pm
The repeated NN-longtime testrun of Lc0 0.23.1 LS12.2 with Kiudee-setting is finished. Impressive progress with long thinking-time, too. https://www.sp-cc.de/nn-longtime-testing.htm
Because much more user use Lc0 nets, jHorthos net and SV nets than LS nets it would be more interesting to investigate the effect of kiudee parameters on these non-LS nets even in the case of short time control too.
Do you not think so?
Hai wrote: ↑Wed Jan 22, 2020 5:13 pm
Do we have now new LC0 default settings?
The tuned settings were done using a mix of short time controls using the T58 (58613) network. These settings should be approximately optimal for all time controls with <20k nodes / move. Even though the CPuct scaling curve is not uniquely determined on slower time controls, it can perform reasonably well on time controls up to <100k nodes, but is likely slightly suboptimal there (though it seemed to perform quite well for T60 on CCC's time control). For 1M+ nodes: no guarantees
I am currently in the process of writing a distributed tuning framework for lc0, which we then could utilize to tune slower time controls than before. These parameters will then likely be better candidates as new defaults, than the currently circulating ones.
Are these setting unique to RTX cards? Would I benefit from trying them on a GTX card?
Thank you
Peter
Hai wrote: ↑Wed Jan 22, 2020 5:13 pm
Do we have now new LC0 default settings?
The tuned settings were done using a mix of short time controls using the T58 (58613) network. These settings should be approximately optimal for all time controls with <20k nodes / move. Even though the CPuct scaling curve is not uniquely determined on slower time controls, it can perform reasonably well on time controls up to <100k nodes, but is likely slightly suboptimal there (though it seemed to perform quite well for T60 on CCC's time control). For 1M+ nodes: no guarantees
I am currently in the process of writing a distributed tuning framework for lc0, which we then could utilize to tune slower time controls than before. These parameters will then likely be better candidates as new defaults, than the currently circulating ones.
Are these setting unique to RTX cards? Would I benefit from trying them on a GTX card?
Thank you
Peter
Yes, node count is important, not the GPU. Anything in the range of 1k - 1000k nodes per move benefits from Kiudee settings, and with most of the nets (well, not too weird ones).
Hai wrote: ↑Wed Jan 22, 2020 5:13 pm
Do we have now new LC0 default settings?
The tuned settings were done using a mix of short time controls using the T58 (58613) network. These settings should be approximately optimal for all time controls with <20k nodes / move. Even though the CPuct scaling curve is not uniquely determined on slower time controls, it can perform reasonably well on time controls up to <100k nodes, but is likely slightly suboptimal there (though it seemed to perform quite well for T60 on CCC's time control). For 1M+ nodes: no guarantees
I am currently in the process of writing a distributed tuning framework for lc0, which we then could utilize to tune slower time controls than before. These parameters will then likely be better candidates as new defaults, than the currently circulating ones.
Are these setting unique to RTX cards? Would I benefit from trying them on a GTX card?
Thank you
Peter
Yes, node count is important, not the GPU. Anything in the range of 1k - 1000k nodes per move benefits from Kiudee settings, and with most of the nets (well, not too weird ones).
Thank you, your explanation helps. I will certainly give it a try.
Peter
There was an earlier suggestion to include changing "Shortsightedness" value. It is suggested in the Github to use a value of 0.02. Someone had tested with 0.05 with good result.
