Lc0: Kiudee setting is very strong

Discussion of anything and everything relating to chess playing software and machines.

Moderators: hgm, Dann Corbit, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
Kiudee
Posts: 27
Joined: Tue Feb 02, 2010 9:12 pm

Re: Lc0: Kiudee setting is very strong

Post by Kiudee » Wed Jan 22, 2020 5:02 pm

Hai wrote:
Wed Jan 22, 2020 4:13 pm
Do we have now new LC0 default settings?
The tuned settings were done using a mix of short time controls using the T58 (58613) network. These settings should be approximately optimal for all time controls with <20k nodes / move. Even though the CPuct scaling curve is not uniquely determined on slower time controls, it can perform reasonably well on time controls up to <100k nodes, but is likely slightly suboptimal there (though it seemed to perform quite well for T60 on CCC's time control). For 1M+ nodes: no guarantees :)

I am currently in the process of writing a distributed tuning framework for lc0, which we then could utilize to tune slower time controls than before. These parameters will then likely be better candidates as new defaults, than the currently circulating ones.

pohl4711
Posts: 1498
Joined: Sat Sep 03, 2011 5:25 am
Location: Berlin, Germany
Contact:

Re: Lc0: Kiudee setting is very strong

Post by pohl4711 » Sun Jan 26, 2020 4:13 pm

Laskos wrote:
Tue Jan 21, 2020 9:23 am
pohl4711 wrote:
Tue Jan 21, 2020 8:05 am
Laskos wrote:
Mon Jan 20, 2020 11:22 pm

Thanks for the info, I can confirm at 30'' + 0.3'' in 200 games from unbalanced openings:

Code: Select all

Score of SV_256x20_1541_Kiudee vs SV_256x20_1541: 52 - 30 - 118  [0.555] 200
Elo difference: 38.37 +/- 30.78
Finished match
On Twitch (potato_chess), 2 testruns with Net 62038 were played (1'+2'' on RTX, which is not that short and 62038 is a bigger Net (24x320)). 200 games each. And with Kiudee, the result was +36 Elo better, than the default setting. So, in each testrun, I have seen, Kiudee scored at least +30 Elo better. Nice!

https://www.twitch.tv/potato_chess

I will repeat my longtime-testrun (8'+5'') of Lc0 0.23.1 LS 12.2 with Kiudee-setting. Lets see, if there is an Elo-gain with long thinkingtime, too...but this will take 6 days from now.

https://www.sp-cc.de/nn-longtime-testing.htm
Good! 1 + 2'' is already Blitz (large increment), so the optimum CPuct value is probably only smoothly increasing to LTC. Thanks for your test at 8' + 5'' in as many as 300 games, a huge task for a single machine. But it is needed, as these Kiudee parameters give a hard to believe boost. In the past I tried to fiddle with Lc0 engine parameters, it was always within even 1 SD error margins, it is hard to have many games at acceptable TC with Leela. Besides that, I do not understand what many of these parameters do precisely, and just blindly trying here and there is useless.

Thanks for posting them, now I will modify Cutechess json file for Lc0 0232 with different nets, all to take these parameters.
The repeated NN-longtime testrun of Lc0 0.23.1 LS12.2 with Kiudee-setting is finished. Impressive progress with long thinking-time, too.

https://www.sp-cc.de/nn-longtime-testing.htm

User avatar
Laskos
Posts: 10949
Joined: Wed Jul 26, 2006 8:21 pm
Full name: Kai Laskos

Re: Lc0: Kiudee setting is very strong

Post by Laskos » Sun Jan 26, 2020 4:57 pm

pohl4711 wrote:
Sun Jan 26, 2020 4:13 pm
Laskos wrote:
Tue Jan 21, 2020 9:23 am
pohl4711 wrote:
Tue Jan 21, 2020 8:05 am
Laskos wrote:
Mon Jan 20, 2020 11:22 pm

Thanks for the info, I can confirm at 30'' + 0.3'' in 200 games from unbalanced openings:

Code: Select all

Score of SV_256x20_1541_Kiudee vs SV_256x20_1541: 52 - 30 - 118  [0.555] 200
Elo difference: 38.37 +/- 30.78
Finished match
On Twitch (potato_chess), 2 testruns with Net 62038 were played (1'+2'' on RTX, which is not that short and 62038 is a bigger Net (24x320)). 200 games each. And with Kiudee, the result was +36 Elo better, than the default setting. So, in each testrun, I have seen, Kiudee scored at least +30 Elo better. Nice!

https://www.twitch.tv/potato_chess

I will repeat my longtime-testrun (8'+5'') of Lc0 0.23.1 LS 12.2 with Kiudee-setting. Lets see, if there is an Elo-gain with long thinkingtime, too...but this will take 6 days from now.

https://www.sp-cc.de/nn-longtime-testing.htm
Good! 1 + 2'' is already Blitz (large increment), so the optimum CPuct value is probably only smoothly increasing to LTC. Thanks for your test at 8' + 5'' in as many as 300 games, a huge task for a single machine. But it is needed, as these Kiudee parameters give a hard to believe boost. In the past I tried to fiddle with Lc0 engine parameters, it was always within even 1 SD error margins, it is hard to have many games at acceptable TC with Leela. Besides that, I do not understand what many of these parameters do precisely, and just blindly trying here and there is useless.

Thanks for posting them, now I will modify Cutechess json file for Lc0 0232 with different nets, all to take these parameters.
The repeated NN-longtime testrun of Lc0 0.23.1 LS12.2 with Kiudee-setting is finished. Impressive progress with long thinking-time, too.

https://www.sp-cc.de/nn-longtime-testing.htm
Excellent, thank you very much! That was a very long run to test at 8 + 5 in 300 games. It confirms my results with T59 128x10 net, which is about 5 times faster in NPS at a similar number of nodes. I will maybe open a separate thread on my results using Kiudee setting settings with the small and fast T59 net. T59 and T60 are trained with very similar parameters as noise etc., but T40 and LS were trained a bit differently, so our results shows that Kiudee settings work for T40 and other nets too to LTC.
My results with T60, T59, T40 nets show about some 40 Elo points improvement at about 50k nodes per move against a variety of strong opponents, and as yours, 30 Elo points at significantly longer TC (200-500k nodes per move range). Now I am testing Kiudee settings with one modification for LTC, CPuct=2.600 instead of 2.147 for LTC. The first results are encouraging, but the test (500 games with T59 net) will be finished in 2 more days. Anyway, I think most with a RTX GPU should switch to Kiudee settings instead of the defaults, the improvement is significant across the ranges of TC and nets.

mwyoung
Posts: 2727
Joined: Wed May 12, 2010 8:00 pm

Re: Lc0: Kiudee setting is very strong

Post by mwyoung » Sun Jan 26, 2020 6:45 pm

pohl4711 wrote:
Sun Jan 26, 2020 4:13 pm
Laskos wrote:
Tue Jan 21, 2020 9:23 am
pohl4711 wrote:
Tue Jan 21, 2020 8:05 am
Laskos wrote:
Mon Jan 20, 2020 11:22 pm

Thanks for the info, I can confirm at 30'' + 0.3'' in 200 games from unbalanced openings:

Code: Select all

Score of SV_256x20_1541_Kiudee vs SV_256x20_1541: 52 - 30 - 118  [0.555] 200
Elo difference: 38.37 +/- 30.78
Finished match
On Twitch (potato_chess), 2 testruns with Net 62038 were played (1'+2'' on RTX, which is not that short and 62038 is a bigger Net (24x320)). 200 games each. And with Kiudee, the result was +36 Elo better, than the default setting. So, in each testrun, I have seen, Kiudee scored at least +30 Elo better. Nice!

https://www.twitch.tv/potato_chess

I will repeat my longtime-testrun (8'+5'') of Lc0 0.23.1 LS 12.2 with Kiudee-setting. Lets see, if there is an Elo-gain with long thinkingtime, too...but this will take 6 days from now.

https://www.sp-cc.de/nn-longtime-testing.htm
Good! 1 + 2'' is already Blitz (large increment), so the optimum CPuct value is probably only smoothly increasing to LTC. Thanks for your test at 8' + 5'' in as many as 300 games, a huge task for a single machine. But it is needed, as these Kiudee parameters give a hard to believe boost. In the past I tried to fiddle with Lc0 engine parameters, it was always within even 1 SD error margins, it is hard to have many games at acceptable TC with Leela. Besides that, I do not understand what many of these parameters do precisely, and just blindly trying here and there is useless.

Thanks for posting them, now I will modify Cutechess json file for Lc0 0232 with different nets, all to take these parameters.
The repeated NN-longtime testrun of Lc0 0.23.1 LS12.2 with Kiudee-setting is finished. Impressive progress with long thinking-time, too.

https://www.sp-cc.de/nn-longtime-testing.htm
I agree. I think the key point is if you are testing with default. You need to understand default in not close to the better settings with the NN. And better settings keep being found.
"The worst thing that can happen to a forum is a running wild attacking moderator(HGM) who is not corrected by the community." - Ed Schröder
But my words like silent raindrops fell. And echoed in the wells of silence.

corres
Posts: 3657
Joined: Wed Nov 18, 2015 10:41 am
Location: hungary

Re: Lc0: Kiudee setting is very strong

Post by corres » Sun Jan 26, 2020 8:50 pm

pohl4711 wrote:
Sun Jan 26, 2020 4:13 pm
The repeated NN-longtime testrun of Lc0 0.23.1 LS12.2 with Kiudee-setting is finished. Impressive progress with long thinking-time, too.
https://www.sp-cc.de/nn-longtime-testing.htm
Because much more user use Lc0 nets, jHorthos net and SV nets than LS nets it would be more interesting to investigate the effect of kiudee parameters on these non-LS nets even in the case of short time control too.
Do you not think so?

fastgm
Posts: 670
Joined: Mon Aug 19, 2013 4:57 pm
Contact:

Re: Lc0: Kiudee setting is very strong

Post by fastgm » Mon Jan 27, 2020 9:17 pm


The repeated NN-longtime testrun of Lc0 0.23.1 LS12.2 with Kiudee-setting is finished. Impressive progress with long thinking-time, too.

https://www.sp-cc.de/nn-longtime-testing.htm
Thanks Stefan for the test! That is good to know.

kasinp
Posts: 216
Joined: Sat Dec 02, 2006 9:47 pm
Location: Toronto
Full name: Peter Kasinski

Re: Lc0: Kiudee setting is very strong

Post by kasinp » Mon Jan 27, 2020 9:50 pm

Kiudee wrote:
Wed Jan 22, 2020 5:02 pm
Hai wrote:
Wed Jan 22, 2020 4:13 pm
Do we have now new LC0 default settings?
The tuned settings were done using a mix of short time controls using the T58 (58613) network. These settings should be approximately optimal for all time controls with <20k nodes / move. Even though the CPuct scaling curve is not uniquely determined on slower time controls, it can perform reasonably well on time controls up to <100k nodes, but is likely slightly suboptimal there (though it seemed to perform quite well for T60 on CCC's time control). For 1M+ nodes: no guarantees :)

I am currently in the process of writing a distributed tuning framework for lc0, which we then could utilize to tune slower time controls than before. These parameters will then likely be better candidates as new defaults, than the currently circulating ones.
Are these setting unique to RTX cards? Would I benefit from trying them on a GTX card?
Thank you
Peter

User avatar
Laskos
Posts: 10949
Joined: Wed Jul 26, 2006 8:21 pm
Full name: Kai Laskos

Re: Lc0: Kiudee setting is very strong

Post by Laskos » Mon Jan 27, 2020 11:38 pm

kasinp wrote:
Mon Jan 27, 2020 9:50 pm
Kiudee wrote:
Wed Jan 22, 2020 5:02 pm
Hai wrote:
Wed Jan 22, 2020 4:13 pm
Do we have now new LC0 default settings?
The tuned settings were done using a mix of short time controls using the T58 (58613) network. These settings should be approximately optimal for all time controls with <20k nodes / move. Even though the CPuct scaling curve is not uniquely determined on slower time controls, it can perform reasonably well on time controls up to <100k nodes, but is likely slightly suboptimal there (though it seemed to perform quite well for T60 on CCC's time control). For 1M+ nodes: no guarantees :)

I am currently in the process of writing a distributed tuning framework for lc0, which we then could utilize to tune slower time controls than before. These parameters will then likely be better candidates as new defaults, than the currently circulating ones.
Are these setting unique to RTX cards? Would I benefit from trying them on a GTX card?
Thank you
Peter
Yes, node count is important, not the GPU. Anything in the range of 1k - 1000k nodes per move benefits from Kiudee settings, and with most of the nets (well, not too weird ones).

kasinp
Posts: 216
Joined: Sat Dec 02, 2006 9:47 pm
Location: Toronto
Full name: Peter Kasinski

Re: Lc0: Kiudee setting is very strong

Post by kasinp » Tue Jan 28, 2020 12:37 am

Laskos wrote:
Mon Jan 27, 2020 11:38 pm
kasinp wrote:
Mon Jan 27, 2020 9:50 pm
Kiudee wrote:
Wed Jan 22, 2020 5:02 pm
Hai wrote:
Wed Jan 22, 2020 4:13 pm
Do we have now new LC0 default settings?
The tuned settings were done using a mix of short time controls using the T58 (58613) network. These settings should be approximately optimal for all time controls with <20k nodes / move. Even though the CPuct scaling curve is not uniquely determined on slower time controls, it can perform reasonably well on time controls up to <100k nodes, but is likely slightly suboptimal there (though it seemed to perform quite well for T60 on CCC's time control). For 1M+ nodes: no guarantees :)

I am currently in the process of writing a distributed tuning framework for lc0, which we then could utilize to tune slower time controls than before. These parameters will then likely be better candidates as new defaults, than the currently circulating ones.
Are these setting unique to RTX cards? Would I benefit from trying them on a GTX card?
Thank you
Peter
Yes, node count is important, not the GPU. Anything in the range of 1k - 1000k nodes per move benefits from Kiudee settings, and with most of the nets (well, not too weird ones).
Thank you, your explanation helps. I will certainly give it a try.
Peter

bmp1974
Posts: 28
Joined: Wed Dec 04, 2019 10:25 am
Full name: Prasanna Bandihole

Re: Lc0: Kiudee setting is very strong

Post by bmp1974 » Tue Jan 28, 2020 4:09 am

There was an earlier suggestion to include changing "Shortsightedness" value. It is suggested in the Github to use a value of 0.02. Someone had tested with 0.05 with good result.

Post Reply