Lc0: Kiudee setting is very strong

Discussion of anything and everything relating to chess playing software and machines.

Moderators: hgm, Dann Corbit, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
Robert Flesher
Posts: 1246
Joined: Tue Aug 18, 2009 1:06 am

Re: Lc0: Kiudee setting is very strong

Post by Robert Flesher » Tue Jan 21, 2020 10:11 pm

pohl4711 wrote:
Mon Jan 20, 2020 1:38 pm
Played 2 Gauntlets a 300 games (30''+300ms Bullet, with my 150 SALC Armageddon openings of my longtime-Testruns (https://www.sp-cc.de/nn-longtime-testing.htm)):
Lc0 default Leelenstein 13 Net vs. Lc0 Kiudee Leelenstein 13 Net and
Lc0 default Leelenstein 13 Net vs. Lc0 LSbinary Leelenstein 13 Net (the binary from josh patreon-site (post from 2019/12/26)

1 Lc0 0.23.2kiudee LS13 vs. Lc0 default LS13: 300 (+180,= 0,-120), 60.0 % (!!!)

2 Lc0 LSbinary LS13 vs. Lc0 default LS13: 300 (+103,= 0,-197), 34.3 %

Conclusions:
Josh-binary is very bad - do not use it!
Lc0 Kiudee is really impressive. 60%-40% means +70 Elo. But mention, that Armageddon (no draws, because all draws are counted as a win for Black) and Bullet-speed spread results, so +40 or +50 Elo seems more realistic. And on discord, some tests with other net-sizes (10x128 and T60) (Leelenstein Size is 20x256) show a measureable Elo gain with Kiudee-setting, too. So, it seems, that the Kiudee-setting should be the new default for Lc0. I will use it from now as default for my Lc0-testings.

Here the Kiudee setting:

CPuct=2.147
Fpu=0.443
PolicyTemperature=1.607
CPuctBase=18368
CPuctFactor=2.815

Thank you for sharing these settings. They are incredible! I am watching crush after crush. Look at this beautiful attack.


jjoshua2
Posts: 99
Joined: Sat Mar 10, 2018 5:16 am

Re: Lc0: Kiudee setting is very strong

Post by jjoshua2 » Wed Jan 22, 2020 1:57 am

The default lc0 params are basically from the a0 paper with a few small modifications that seemed to help with GPU and such. Lc0 team hasn't used defaults at CCC or TCEC for a long time, because they aren't very good at LTC since they don't widen up search often to play AB engines good at tactics and still fall to a lot of blunders even with a millions of nodes (not too much you can do but settings do help some). Default settings are even worse at very fast TC.

The Leelenstein binary includes even wider setting than lc0 uses at TCEC. It was tuned against SF at 150k nodes / move as that is the minimum ever gets at CCC with 60s+1 and 4x2080 ti. It is definitely much worse at 1000 nodes and even 10s / game on a 2080 ti OC. And also probably not optimal in self play or similar. But you can enter Kiudee search params into Leelenstein binary and it will work well in many situations, and get to make use of some other features.

Leelenstein also defaults with some trade penalty which penalizes trading down. Obviously this hurts a bit in self play, but in round robin tournaments even of top 6 engines it gains about 15 elo it seems, and doesn't seem to hurt even against a similar strength SF. But this is a UCI parameter you can set to zero and disable as well. Correspondence players like to analyze with two different settings to get two different PV lines with trading down or not...

Leelenstein also has some certainty propogation and mate scores so it can play into forced wins and announce them (and prune forced loss subtrees). And in some situations it prunes drawing lines after just 2 reps instead of 3 like Leela.

So it might make sense to use the same search params and set trade penalty to zero to compare the intrinsic binary features, since you can enter a tune for your TC and opponent into any binary. Interestingly though, the default trade penalty amount increases the ICCF agreement of moves played (over none) in my testing.

EDIT: It will be interesting to see as results start to come in with Kiudee's tune at longer TC against Stockfish.

Hai
Posts: 548
Joined: Sun Aug 04, 2013 11:19 am

Re: Lc0: Kiudee setting is very strong

Post by Hai » Wed Jan 22, 2020 4:13 pm

Do we have now new LC0 default settings?

Kiudee
Posts: 27
Joined: Tue Feb 02, 2010 9:12 pm

Re: Lc0: Kiudee setting is very strong

Post by Kiudee » Wed Jan 22, 2020 5:02 pm

Hai wrote:
Wed Jan 22, 2020 4:13 pm
Do we have now new LC0 default settings?
The tuned settings were done using a mix of short time controls using the T58 (58613) network. These settings should be approximately optimal for all time controls with <20k nodes / move. Even though the CPuct scaling curve is not uniquely determined on slower time controls, it can perform reasonably well on time controls up to <100k nodes, but is likely slightly suboptimal there (though it seemed to perform quite well for T60 on CCC's time control). For 1M+ nodes: no guarantees :)

I am currently in the process of writing a distributed tuning framework for lc0, which we then could utilize to tune slower time controls than before. These parameters will then likely be better candidates as new defaults, than the currently circulating ones.

pohl4711
Posts: 1541
Joined: Sat Sep 03, 2011 5:25 am
Location: Berlin, Germany
Contact:

Re: Lc0: Kiudee setting is very strong

Post by pohl4711 » Sun Jan 26, 2020 4:13 pm

Laskos wrote:
Tue Jan 21, 2020 9:23 am
pohl4711 wrote:
Tue Jan 21, 2020 8:05 am
Laskos wrote:
Mon Jan 20, 2020 11:22 pm

Thanks for the info, I can confirm at 30'' + 0.3'' in 200 games from unbalanced openings:

Code: Select all

Score of SV_256x20_1541_Kiudee vs SV_256x20_1541: 52 - 30 - 118  [0.555] 200
Elo difference: 38.37 +/- 30.78
Finished match
On Twitch (potato_chess), 2 testruns with Net 62038 were played (1'+2'' on RTX, which is not that short and 62038 is a bigger Net (24x320)). 200 games each. And with Kiudee, the result was +36 Elo better, than the default setting. So, in each testrun, I have seen, Kiudee scored at least +30 Elo better. Nice!

https://www.twitch.tv/potato_chess

I will repeat my longtime-testrun (8'+5'') of Lc0 0.23.1 LS 12.2 with Kiudee-setting. Lets see, if there is an Elo-gain with long thinkingtime, too...but this will take 6 days from now.

https://www.sp-cc.de/nn-longtime-testing.htm
Good! 1 + 2'' is already Blitz (large increment), so the optimum CPuct value is probably only smoothly increasing to LTC. Thanks for your test at 8' + 5'' in as many as 300 games, a huge task for a single machine. But it is needed, as these Kiudee parameters give a hard to believe boost. In the past I tried to fiddle with Lc0 engine parameters, it was always within even 1 SD error margins, it is hard to have many games at acceptable TC with Leela. Besides that, I do not understand what many of these parameters do precisely, and just blindly trying here and there is useless.

Thanks for posting them, now I will modify Cutechess json file for Lc0 0232 with different nets, all to take these parameters.
The repeated NN-longtime testrun of Lc0 0.23.1 LS12.2 with Kiudee-setting is finished. Impressive progress with long thinking-time, too.

https://www.sp-cc.de/nn-longtime-testing.htm

User avatar
Laskos
Posts: 10949
Joined: Wed Jul 26, 2006 8:21 pm
Full name: Kai Laskos

Re: Lc0: Kiudee setting is very strong

Post by Laskos » Sun Jan 26, 2020 4:57 pm

pohl4711 wrote:
Sun Jan 26, 2020 4:13 pm
Laskos wrote:
Tue Jan 21, 2020 9:23 am
pohl4711 wrote:
Tue Jan 21, 2020 8:05 am
Laskos wrote:
Mon Jan 20, 2020 11:22 pm

Thanks for the info, I can confirm at 30'' + 0.3'' in 200 games from unbalanced openings:

Code: Select all

Score of SV_256x20_1541_Kiudee vs SV_256x20_1541: 52 - 30 - 118  [0.555] 200
Elo difference: 38.37 +/- 30.78
Finished match
On Twitch (potato_chess), 2 testruns with Net 62038 were played (1'+2'' on RTX, which is not that short and 62038 is a bigger Net (24x320)). 200 games each. And with Kiudee, the result was +36 Elo better, than the default setting. So, in each testrun, I have seen, Kiudee scored at least +30 Elo better. Nice!

https://www.twitch.tv/potato_chess

I will repeat my longtime-testrun (8'+5'') of Lc0 0.23.1 LS 12.2 with Kiudee-setting. Lets see, if there is an Elo-gain with long thinkingtime, too...but this will take 6 days from now.

https://www.sp-cc.de/nn-longtime-testing.htm
Good! 1 + 2'' is already Blitz (large increment), so the optimum CPuct value is probably only smoothly increasing to LTC. Thanks for your test at 8' + 5'' in as many as 300 games, a huge task for a single machine. But it is needed, as these Kiudee parameters give a hard to believe boost. In the past I tried to fiddle with Lc0 engine parameters, it was always within even 1 SD error margins, it is hard to have many games at acceptable TC with Leela. Besides that, I do not understand what many of these parameters do precisely, and just blindly trying here and there is useless.

Thanks for posting them, now I will modify Cutechess json file for Lc0 0232 with different nets, all to take these parameters.
The repeated NN-longtime testrun of Lc0 0.23.1 LS12.2 with Kiudee-setting is finished. Impressive progress with long thinking-time, too.

https://www.sp-cc.de/nn-longtime-testing.htm
Excellent, thank you very much! That was a very long run to test at 8 + 5 in 300 games. It confirms my results with T59 128x10 net, which is about 5 times faster in NPS at a similar number of nodes. I will maybe open a separate thread on my results using Kiudee setting settings with the small and fast T59 net. T59 and T60 are trained with very similar parameters as noise etc., but T40 and LS were trained a bit differently, so our results shows that Kiudee settings work for T40 and other nets too to LTC.
My results with T60, T59, T40 nets show about some 40 Elo points improvement at about 50k nodes per move against a variety of strong opponents, and as yours, 30 Elo points at significantly longer TC (200-500k nodes per move range). Now I am testing Kiudee settings with one modification for LTC, CPuct=2.600 instead of 2.147 for LTC. The first results are encouraging, but the test (500 games with T59 net) will be finished in 2 more days. Anyway, I think most with a RTX GPU should switch to Kiudee settings instead of the defaults, the improvement is significant across the ranges of TC and nets.

mwyoung
Posts: 2727
Joined: Wed May 12, 2010 8:00 pm

Re: Lc0: Kiudee setting is very strong

Post by mwyoung » Sun Jan 26, 2020 6:45 pm

pohl4711 wrote:
Sun Jan 26, 2020 4:13 pm
Laskos wrote:
Tue Jan 21, 2020 9:23 am
pohl4711 wrote:
Tue Jan 21, 2020 8:05 am
Laskos wrote:
Mon Jan 20, 2020 11:22 pm

Thanks for the info, I can confirm at 30'' + 0.3'' in 200 games from unbalanced openings:

Code: Select all

Score of SV_256x20_1541_Kiudee vs SV_256x20_1541: 52 - 30 - 118  [0.555] 200
Elo difference: 38.37 +/- 30.78
Finished match
On Twitch (potato_chess), 2 testruns with Net 62038 were played (1'+2'' on RTX, which is not that short and 62038 is a bigger Net (24x320)). 200 games each. And with Kiudee, the result was +36 Elo better, than the default setting. So, in each testrun, I have seen, Kiudee scored at least +30 Elo better. Nice!

https://www.twitch.tv/potato_chess

I will repeat my longtime-testrun (8'+5'') of Lc0 0.23.1 LS 12.2 with Kiudee-setting. Lets see, if there is an Elo-gain with long thinkingtime, too...but this will take 6 days from now.

https://www.sp-cc.de/nn-longtime-testing.htm
Good! 1 + 2'' is already Blitz (large increment), so the optimum CPuct value is probably only smoothly increasing to LTC. Thanks for your test at 8' + 5'' in as many as 300 games, a huge task for a single machine. But it is needed, as these Kiudee parameters give a hard to believe boost. In the past I tried to fiddle with Lc0 engine parameters, it was always within even 1 SD error margins, it is hard to have many games at acceptable TC with Leela. Besides that, I do not understand what many of these parameters do precisely, and just blindly trying here and there is useless.

Thanks for posting them, now I will modify Cutechess json file for Lc0 0232 with different nets, all to take these parameters.
The repeated NN-longtime testrun of Lc0 0.23.1 LS12.2 with Kiudee-setting is finished. Impressive progress with long thinking-time, too.

https://www.sp-cc.de/nn-longtime-testing.htm
I agree. I think the key point is if you are testing with default. You need to understand default in not close to the better settings with the NN. And better settings keep being found.
"The worst thing that can happen to a forum is a running wild attacking moderator(HGM) who is not corrected by the community." - Ed Schröder
But my words like silent raindrops fell. And echoed in the wells of silence.

corres
Posts: 3657
Joined: Wed Nov 18, 2015 10:41 am
Location: hungary

Re: Lc0: Kiudee setting is very strong

Post by corres » Sun Jan 26, 2020 8:50 pm

pohl4711 wrote:
Sun Jan 26, 2020 4:13 pm
The repeated NN-longtime testrun of Lc0 0.23.1 LS12.2 with Kiudee-setting is finished. Impressive progress with long thinking-time, too.
https://www.sp-cc.de/nn-longtime-testing.htm
Because much more user use Lc0 nets, jHorthos net and SV nets than LS nets it would be more interesting to investigate the effect of kiudee parameters on these non-LS nets even in the case of short time control too.
Do you not think so?

fastgm
Posts: 700
Joined: Mon Aug 19, 2013 4:57 pm
Contact:

Re: Lc0: Kiudee setting is very strong

Post by fastgm » Mon Jan 27, 2020 9:17 pm


The repeated NN-longtime testrun of Lc0 0.23.1 LS12.2 with Kiudee-setting is finished. Impressive progress with long thinking-time, too.

https://www.sp-cc.de/nn-longtime-testing.htm
Thanks Stefan for the test! That is good to know.

kasinp
Posts: 217
Joined: Sat Dec 02, 2006 9:47 pm
Location: Toronto
Full name: Peter Kasinski

Re: Lc0: Kiudee setting is very strong

Post by kasinp » Mon Jan 27, 2020 9:50 pm

Kiudee wrote:
Wed Jan 22, 2020 5:02 pm
Hai wrote:
Wed Jan 22, 2020 4:13 pm
Do we have now new LC0 default settings?
The tuned settings were done using a mix of short time controls using the T58 (58613) network. These settings should be approximately optimal for all time controls with <20k nodes / move. Even though the CPuct scaling curve is not uniquely determined on slower time controls, it can perform reasonably well on time controls up to <100k nodes, but is likely slightly suboptimal there (though it seemed to perform quite well for T60 on CCC's time control). For 1M+ nodes: no guarantees :)

I am currently in the process of writing a distributed tuning framework for lc0, which we then could utilize to tune slower time controls than before. These parameters will then likely be better candidates as new defaults, than the currently circulating ones.
Are these setting unique to RTX cards? Would I benefit from trying them on a GTX card?
Thank you
Peter

Post Reply