Lc0: Kiudee setting is very strong

[Robert Flesher]

Played 2 Gauntlets a 300 games (30''+300ms Bullet, with my 150 SALC Armageddon openings of my longtime-Testruns (https://www.sp-cc.de/nn-longtime-testing.htm)):
Lc0 default Leelenstein 13 Net vs. Lc0 Kiudee Leelenstein 13 Net and
Lc0 default Leelenstein 13 Net vs. Lc0 LSbinary Leelenstein 13 Net (the binary from josh patreon-site (post from 2019/12/26)

1 Lc0 0.23.2kiudee LS13 vs. Lc0 default LS13: 300 (+180,= 0,-120), 60.0 % (!!!)

2 Lc0 LSbinary LS13 vs. Lc0 default LS13: 300 (+103,= 0,-197), 34.3 %

Conclusions:
Josh-binary is very bad - do not use it!
Lc0 Kiudee is really impressive. 60%-40% means +70 Elo. But mention, that Armageddon (no draws, because all draws are counted as a win for Black) and Bullet-speed spread results, so +40 or +50 Elo seems more realistic. And on discord, some tests with other net-sizes (10x128 and T60) (Leelenstein Size is 20x256) show a measureable Elo gain with Kiudee-setting, too. So, it seems, that the Kiudee-setting should be the new default for Lc0. I will use it from now as default for my Lc0-testings.

Here the Kiudee setting:

CPuct=2.147
Fpu=0.443
PolicyTemperature=1.607
CPuctBase=18368
CPuctFactor=2.815

Thank you for sharing these settings. They are incredible! I am watching crush after crush. Look at this beautiful attack.

[pgn] [Event "TINCAN, Blitz 3m+7s"]
[Site "TINCAN"]
[Date "2020.01.21"]
[Round "3"]
[White "Lc0 v0.23.2+git.c8d9095 T60"]
[Black "ShashChess 10 64 POPCNT"]
[Result "1-0"]
[ECO "E26"]
[PlyCount "108"]

1. d4 Nf6 2. c4 e6 3. Nc3 Bb4 4. a3 Bxc3+ 5. bxc3 c5 6. e3 b6 7. Bd3 Bb7 8. f3
Nc6 9. Ne2 d6 10. e4 Na5 11. O-O Ba6 12. f4 O-O 13. f5 Bxc4 14. Bg5 Bb3 15. Qc1
cxd4 16. Ng3 h6 17. Bxh6 Ng4 18. Bxg7 Qh4 19. h3 Ne3 20. Rf3 Bd1 21. Rxe3 dxe3
22. Bxf8 Qxg3 23. Bxd6 Qf2+ 24. Kh2 Bf3 25. Qg1 Rd8 26. e5 Qxg1+ 27. Kxg1 Bd5
28. Re1 exf5 29. Bxf5 Nc4 30. Be7 Rb8 31. Bf6 Be6 32. Bd3 Rc8 33. Bxc4 Rxc4 34.
Rxe3 Ra4 35. Rg3+ Kh7 36. Rg7+ Kh6 37. g4 Rc4 38. g5+ Kh5 39. g6 Rc8 40. Kf2
Kh6 41. gxf7 Rf8 42. Ke3 Rxf7 43. Rxf7 Bxf7 44. Kd4 Kg6 45. c4 Be8 46. Bd8 Kf5
47. Bc7 Ke6 48. Bb8 b5 49. cxb5 Bxb5 50. Bxa7 Ba4 51. Ke4 Bd1 52. Bb8 Bc2+ 53.
Kf4 Kd5 54. Bd6 Kc6 1-0 [/pgn]

[jjoshua2]

The default lc0 params are basically from the a0 paper with a few small modifications that seemed to help with GPU and such. Lc0 team hasn't used defaults at CCC or TCEC for a long time, because they aren't very good at LTC since they don't widen up search often to play AB engines good at tactics and still fall to a lot of blunders even with a millions of nodes (not too much you can do but settings do help some). Default settings are even worse at very fast TC.

The Leelenstein binary includes even wider setting than lc0 uses at TCEC. It was tuned against SF at 150k nodes / move as that is the minimum ever gets at CCC with 60s+1 and 4x2080 ti. It is definitely much worse at 1000 nodes and even 10s / game on a 2080 ti OC. And also probably not optimal in self play or similar. But you can enter Kiudee search params into Leelenstein binary and it will work well in many situations, and get to make use of some other features.

Leelenstein also defaults with some trade penalty which penalizes trading down. Obviously this hurts a bit in self play, but in round robin tournaments even of top 6 engines it gains about 15 elo it seems, and doesn't seem to hurt even against a similar strength SF. But this is a UCI parameter you can set to zero and disable as well. Correspondence players like to analyze with two different settings to get two different PV lines with trading down or not...

Leelenstein also has some certainty propogation and mate scores so it can play into forced wins and announce them (and prune forced loss subtrees). And in some situations it prunes drawing lines after just 2 reps instead of 3 like Leela.

So it might make sense to use the same search params and set trade penalty to zero to compare the intrinsic binary features, since you can enter a tune for your TC and opponent into any binary. Interestingly though, the default trade penalty amount increases the ICCF agreement of moves played (over none) in my testing.

EDIT: It will be interesting to see as results start to come in with Kiudee's tune at longer TC against Stockfish.

[Hai]

Do we have now new LC0 default settings?

[Kiudee]

Do we have now new LC0 default settings?

The tuned settings were done using a mix of short time controls using the T58 (58613) network. These settings should be approximately optimal for all time controls with <20k nodes / move. Even though the CPuct scaling curve is not uniquely determined on slower time controls, it can perform reasonably well on time controls up to <100k nodes, but is likely slightly suboptimal there (though it seemed to perform quite well for T60 on CCC's time control). For 1M+ nodes: no guarantees

I am currently in the process of writing a distributed tuning framework for lc0, which we then could utilize to tune slower time controls than before. These parameters will then likely be better candidates as new defaults, than the currently circulating ones.

[pohl4711]

Thanks for the info, I can confirm at 30'' + 0.3'' in 200 games from unbalanced openings:

Code: Select all

Score of SV_256x20_1541_Kiudee vs SV_256x20_1541: 52 - 30 - 118  [0.555] 200
Elo difference: 38.37 +/- 30.78
Finished match

On Twitch (potato_chess), 2 testruns with Net 62038 were played (1'+2'' on RTX, which is not that short and 62038 is a bigger Net (24x320)). 200 games each. And with Kiudee, the result was +36 Elo better, than the default setting. So, in each testrun, I have seen, Kiudee scored at least +30 Elo better. Nice!



I will repeat my longtime-testrun (8'+5'') of Lc0 0.23.1 LS 12.2 with Kiudee-setting. Lets see, if there is an Elo-gain with long thinkingtime, too...but this will take 6 days from now.

https://www.sp-cc.de/nn-longtime-testing.htm

Good! 1 + 2'' is already Blitz (large increment), so the optimum CPuct value is probably only smoothly increasing to LTC. Thanks for your test at 8' + 5'' in as many as 300 games, a huge task for a single machine. But it is needed, as these Kiudee parameters give a hard to believe boost. In the past I tried to fiddle with Lc0 engine parameters, it was always within even 1 SD error margins, it is hard to have many games at acceptable TC with Leela. Besides that, I do not understand what many of these parameters do precisely, and just blindly trying here and there is useless.

Thanks for posting them, now I will modify Cutechess json file for Lc0 0232 with different nets, all to take these parameters.

The repeated NN-longtime testrun of Lc0 0.23.1 LS12.2 with Kiudee-setting is finished. Impressive progress with long thinking-time, too.

https://www.sp-cc.de/nn-longtime-testing.htm

[Laskos]

Thanks for the info, I can confirm at 30'' + 0.3'' in 200 games from unbalanced openings:

Code: Select all

Score of SV_256x20_1541_Kiudee vs SV_256x20_1541: 52 - 30 - 118  [0.555] 200
Elo difference: 38.37 +/- 30.78
Finished match

On Twitch (potato_chess), 2 testruns with Net 62038 were played (1'+2'' on RTX, which is not that short and 62038 is a bigger Net (24x320)). 200 games each. And with Kiudee, the result was +36 Elo better, than the default setting. So, in each testrun, I have seen, Kiudee scored at least +30 Elo better. Nice!



I will repeat my longtime-testrun (8'+5'') of Lc0 0.23.1 LS 12.2 with Kiudee-setting. Lets see, if there is an Elo-gain with long thinkingtime, too...but this will take 6 days from now.

https://www.sp-cc.de/nn-longtime-testing.htm

Good! 1 + 2'' is already Blitz (large increment), so the optimum CPuct value is probably only smoothly increasing to LTC. Thanks for your test at 8' + 5'' in as many as 300 games, a huge task for a single machine. But it is needed, as these Kiudee parameters give a hard to believe boost. In the past I tried to fiddle with Lc0 engine parameters, it was always within even 1 SD error margins, it is hard to have many games at acceptable TC with Leela. Besides that, I do not understand what many of these parameters do precisely, and just blindly trying here and there is useless.

Thanks for posting them, now I will modify Cutechess json file for Lc0 0232 with different nets, all to take these parameters.

The repeated NN-longtime testrun of Lc0 0.23.1 LS12.2 with Kiudee-setting is finished. Impressive progress with long thinking-time, too.

https://www.sp-cc.de/nn-longtime-testing.htm

Excellent, thank you very much! That was a very long run to test at 8 + 5 in 300 games. It confirms my results with T59 128x10 net, which is about 5 times faster in NPS at a similar number of nodes. I will maybe open a separate thread on my results using Kiudee setting settings with the small and fast T59 net. T59 and T60 are trained with very similar parameters as noise etc., but T40 and LS were trained a bit differently, so our results shows that Kiudee settings work for T40 and other nets too to LTC.
My results with T60, T59, T40 nets show about some 40 Elo points improvement at about 50k nodes per move against a variety of strong opponents, and as yours, 30 Elo points at significantly longer TC (200-500k nodes per move range). Now I am testing Kiudee settings with one modification for LTC, CPuct=2.600 instead of 2.147 for LTC. The first results are encouraging, but the test (500 games with T59 net) will be finished in 2 more days. Anyway, I think most with a RTX GPU should switch to Kiudee settings instead of the defaults, the improvement is significant across the ranges of TC and nets.

[mwyoung]

Thanks for the info, I can confirm at 30'' + 0.3'' in 200 games from unbalanced openings:

Code: Select all

Score of SV_256x20_1541_Kiudee vs SV_256x20_1541: 52 - 30 - 118  [0.555] 200
Elo difference: 38.37 +/- 30.78
Finished match

On Twitch (potato_chess), 2 testruns with Net 62038 were played (1'+2'' on RTX, which is not that short and 62038 is a bigger Net (24x320)). 200 games each. And with Kiudee, the result was +36 Elo better, than the default setting. So, in each testrun, I have seen, Kiudee scored at least +30 Elo better. Nice!



I will repeat my longtime-testrun (8'+5'') of Lc0 0.23.1 LS 12.2 with Kiudee-setting. Lets see, if there is an Elo-gain with long thinkingtime, too...but this will take 6 days from now.

https://www.sp-cc.de/nn-longtime-testing.htm

Good! 1 + 2'' is already Blitz (large increment), so the optimum CPuct value is probably only smoothly increasing to LTC. Thanks for your test at 8' + 5'' in as many as 300 games, a huge task for a single machine. But it is needed, as these Kiudee parameters give a hard to believe boost. In the past I tried to fiddle with Lc0 engine parameters, it was always within even 1 SD error margins, it is hard to have many games at acceptable TC with Leela. Besides that, I do not understand what many of these parameters do precisely, and just blindly trying here and there is useless.

Thanks for posting them, now I will modify Cutechess json file for Lc0 0232 with different nets, all to take these parameters.

The repeated NN-longtime testrun of Lc0 0.23.1 LS12.2 with Kiudee-setting is finished. Impressive progress with long thinking-time, too.

https://www.sp-cc.de/nn-longtime-testing.htm

I agree. I think the key point is if you are testing with default. You need to understand default in not close to the better settings with the NN. And better settings keep being found.

[corres]

The repeated NN-longtime testrun of Lc0 0.23.1 LS12.2 with Kiudee-setting is finished. Impressive progress with long thinking-time, too.
https://www.sp-cc.de/nn-longtime-testing.htm

Because much more user use Lc0 nets, jHorthos net and SV nets than LS nets it would be more interesting to investigate the effect of kiudee parameters on these non-LS nets even in the case of short time control too.
Do you not think so?

[fastgm]

The repeated NN-longtime testrun of Lc0 0.23.1 LS12.2 with Kiudee-setting is finished. Impressive progress with long thinking-time, too.

https://www.sp-cc.de/nn-longtime-testing.htm

Thanks Stefan for the test! That is good to know.

[kasinp]

Do we have now new LC0 default settings?

The tuned settings were done using a mix of short time controls using the T58 (58613) network. These settings should be approximately optimal for all time controls with <20k nodes / move. Even though the CPuct scaling curve is not uniquely determined on slower time controls, it can perform reasonably well on time controls up to <100k nodes, but is likely slightly suboptimal there (though it seemed to perform quite well for T60 on CCC's time control). For 1M+ nodes: no guarantees

I am currently in the process of writing a distributed tuning framework for lc0, which we then could utilize to tune slower time controls than before. These parameters will then likely be better candidates as new defaults, than the currently circulating ones.

Are these setting unique to RTX cards? Would I benefit from trying them on a GTX card?
Thank you
Peter