Lc0: Kiudee setting is very strong

Discussion of anything and everything relating to chess playing software and machines.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
corres
Posts: 2089
Joined: Wed Nov 18, 2015 10:41 am
Location: hungary

Re: Lc0: Kiudee setting is very strong

Post by corres » Mon Jan 20, 2020 11:58 pm

Laskos wrote:
Mon Jan 20, 2020 11:49 pm
corres wrote:
Mon Jan 20, 2020 11:47 pm
Laskos wrote:
Mon Jan 20, 2020 11:22 pm
...
Thanks for the info, I can confirm at 30'' + 0.3'' in 200 games from unbalanced openings:

Code: Select all

Score of SV_256x20_1541_Kiudee vs SV_256x20_1541: 52 - 30 - 118  [0.555] 200
Elo difference: 38.37 +/- 30.78
Finished match
The engine is Lc0 v0.23.2.
With unbalanced openings in paired games one should use the pentanomial error margins, which are smaller than the usual trinomial ones. Here is how a result looks in my case:
Image
So, all in all about 40 +/- 20 (95% confidence) Elo points improvement, which is amazing. I guess the optimal CPuct will increase with longer time control, but I have no resources to tune it to various TC.
Again, 40 Elo points gain is amazing.
Is our reported result an "Armageddon" result or a normal, really experienced result?
Using unbalanced start positions is not equivalent to using "Armageddon" method to modify the result of games.
No, regular 3-mover human openings, somewhat unbalanced to avoid 90% draw rates. From games of above 2200 FIDE Elo points humans.
Without "Armageddon"?

User avatar
Laskos
Posts: 9843
Joined: Wed Jul 26, 2006 8:21 pm
Full name: Kai Laskos

Re: Lc0: Kiudee setting is very strong

Post by Laskos » Tue Jan 21, 2020 12:01 am

corres wrote:
Mon Jan 20, 2020 11:58 pm
Laskos wrote:
Mon Jan 20, 2020 11:49 pm
corres wrote:
Mon Jan 20, 2020 11:47 pm
Laskos wrote:
Mon Jan 20, 2020 11:22 pm
...
Thanks for the info, I can confirm at 30'' + 0.3'' in 200 games from unbalanced openings:

Code: Select all

Score of SV_256x20_1541_Kiudee vs SV_256x20_1541: 52 - 30 - 118  [0.555] 200
Elo difference: 38.37 +/- 30.78
Finished match
The engine is Lc0 v0.23.2.
With unbalanced openings in paired games one should use the pentanomial error margins, which are smaller than the usual trinomial ones. Here is how a result looks in my case:
Image
So, all in all about 40 +/- 20 (95% confidence) Elo points improvement, which is amazing. I guess the optimal CPuct will increase with longer time control, but I have no resources to tune it to various TC.
Again, 40 Elo points gain is amazing.
Is our reported result an "Armageddon" result or a normal, really experienced result?
Using unbalanced start positions is not equivalent to using "Armageddon" method to modify the result of games.
No, regular 3-mover human openings, somewhat unbalanced to avoid 90% draw rates. From games of above 2200 FIDE Elo points humans.
Without "Armageddon"?
No, no, normal games and scoring. Armageddon is a bit different chess, I am not testing with it as of yet. Unbalanced openings are almost as good as resolving power goes, but it's actual chess and scoring.

corres
Posts: 2089
Joined: Wed Nov 18, 2015 10:41 am
Location: hungary

Re: Lc0: Kiudee setting is very strong

Post by corres » Tue Jan 21, 2020 12:16 am

Laskos wrote:
Tue Jan 21, 2020 12:01 am
corres wrote:
Mon Jan 20, 2020 11:58 pm
Laskos wrote:
Mon Jan 20, 2020 11:49 pm
corres wrote:
Mon Jan 20, 2020 11:47 pm
Laskos wrote:
Mon Jan 20, 2020 11:22 pm
...
Again, 40 Elo points gain is amazing.
Is our reported result an "Armageddon" result or a normal, really experienced result?
Using unbalanced start positions is not equivalent to using "Armageddon" method to modify the result of games.
No, regular 3-mover human openings, somewhat unbalanced to avoid 90% draw rates. From games of above 2200 FIDE Elo points humans.
Without "Armageddon"?
No, no, normal games and scoring. Armageddon is a bit different chess, I am not testing with it as of yet. Unbalanced openings are almost as good as resolving power goes, but it's actual chess and scoring.
Thanks for the answers.

bmp1974
Posts: 14
Joined: Wed Dec 04, 2019 10:25 am
Full name: Prasanna Bandihole

Re: Lc0: Kiudee setting is very strong

Post by bmp1974 » Tue Jan 21, 2020 7:34 am

Has anyone observed improvement with regard to endgame play by Lc0? Does the Kiudee settings avoid Lc0 trolling in endgames?
My prediction is that, in coming few months, with T60 NN reaching elo 3000 mark and with settings that help improve endgame play quality, Lc0 will be unbeatable!

pohl4711
Posts: 1164
Joined: Sat Sep 03, 2011 5:25 am
Location: Berlin, Germany
Contact:

Re: Lc0: Kiudee setting is very strong

Post by pohl4711 » Tue Jan 21, 2020 8:05 am

Laskos wrote:
Mon Jan 20, 2020 11:22 pm

Thanks for the info, I can confirm at 30'' + 0.3'' in 200 games from unbalanced openings:

Code: Select all

Score of SV_256x20_1541_Kiudee vs SV_256x20_1541: 52 - 30 - 118  [0.555] 200
Elo difference: 38.37 +/- 30.78
Finished match
On Twitch (potato_chess), 2 testruns with Net 62038 were played (1'+2'' on RTX, which is not that short and 62038 is a bigger Net (24x320)). 200 games each. And with Kiudee, the result was +36 Elo better, than the default setting. So, in each testrun, I have seen, Kiudee scored at least +30 Elo better. Nice!

https://www.twitch.tv/potato_chess

I will repeat my longtime-testrun (8'+5'') of Lc0 0.23.1 LS 12.2 with Kiudee-setting. Lets see, if there is an Elo-gain with long thinkingtime, too...but this will take 6 days from now.

https://www.sp-cc.de/nn-longtime-testing.htm

User avatar
Laskos
Posts: 9843
Joined: Wed Jul 26, 2006 8:21 pm
Full name: Kai Laskos

Re: Lc0: Kiudee setting is very strong

Post by Laskos » Tue Jan 21, 2020 9:23 am

pohl4711 wrote:
Tue Jan 21, 2020 8:05 am
Laskos wrote:
Mon Jan 20, 2020 11:22 pm

Thanks for the info, I can confirm at 30'' + 0.3'' in 200 games from unbalanced openings:

Code: Select all

Score of SV_256x20_1541_Kiudee vs SV_256x20_1541: 52 - 30 - 118  [0.555] 200
Elo difference: 38.37 +/- 30.78
Finished match
On Twitch (potato_chess), 2 testruns with Net 62038 were played (1'+2'' on RTX, which is not that short and 62038 is a bigger Net (24x320)). 200 games each. And with Kiudee, the result was +36 Elo better, than the default setting. So, in each testrun, I have seen, Kiudee scored at least +30 Elo better. Nice!

https://www.twitch.tv/potato_chess

I will repeat my longtime-testrun (8'+5'') of Lc0 0.23.1 LS 12.2 with Kiudee-setting. Lets see, if there is an Elo-gain with long thinkingtime, too...but this will take 6 days from now.

https://www.sp-cc.de/nn-longtime-testing.htm
Good! 1 + 2'' is already Blitz (large increment), so the optimum CPuct value is probably only smoothly increasing to LTC. Thanks for your test at 8' + 5'' in as many as 300 games, a huge task for a single machine. But it is needed, as these Kiudee parameters give a hard to believe boost. In the past I tried to fiddle with Lc0 engine parameters, it was always within even 1 SD error margins, it is hard to have many games at acceptable TC with Leela. Besides that, I do not understand what many of these parameters do precisely, and just blindly trying here and there is useless.

Thanks for posting them, now I will modify Cutechess json file for Lc0 0232 with different nets, all to take these parameters.

pohl4711
Posts: 1164
Joined: Sat Sep 03, 2011 5:25 am
Location: Berlin, Germany
Contact:

Re: Lc0: Kiudee setting is very strong

Post by pohl4711 » Tue Jan 21, 2020 9:40 am

Laskos wrote:
Tue Jan 21, 2020 9:23 am


Good! 1 + 2'' is already Blitz (large increment), so the optimum CPuct value is probably only smoothly increasing to LTC. Thanks for your test at 8' + 5'' in as many as 300 games, a huge task for a single machine. But it is needed, as these Kiudee parameters give a hard to believe boost. In the past I tried to fiddle with Lc0 engine parameters, it was always within even 1 SD error margins, it is hard to have many games at acceptable TC with Leela. Besides that, I do not understand what many of these parameters do precisely, and just blindly trying here and there is useless.
Kiudee settings are done by tuning-runs (Kiudee is the nickname of the author on discord). At the moment, he tunes the ShortSightedness parameter (all other parameters are set to the known Kiudee-setting, we test here).


https://discordapp.com/channels/4254194 ... 8982059008
"Started a small tuning run for ShortSightedness. Same TC as my other tunes, with kiudee-bonus applied. Network: 58613.
Is there a plausible reason for why there is a bump in the middle?"

https://cdn.discordapp.com/attachments/ ... 26-339.png

User avatar
Laskos
Posts: 9843
Joined: Wed Jul 26, 2006 8:21 pm
Full name: Kai Laskos

Re: Lc0: Kiudee setting is very strong

Post by Laskos » Tue Jan 21, 2020 12:11 pm

pohl4711 wrote:
Tue Jan 21, 2020 9:40 am
Laskos wrote:
Tue Jan 21, 2020 9:23 am


Good! 1 + 2'' is already Blitz (large increment), so the optimum CPuct value is probably only smoothly increasing to LTC. Thanks for your test at 8' + 5'' in as many as 300 games, a huge task for a single machine. But it is needed, as these Kiudee parameters give a hard to believe boost. In the past I tried to fiddle with Lc0 engine parameters, it was always within even 1 SD error margins, it is hard to have many games at acceptable TC with Leela. Besides that, I do not understand what many of these parameters do precisely, and just blindly trying here and there is useless.
Kiudee settings are done by tuning-runs (Kiudee is the nickname of the author on discord). At the moment, he tunes the ShortSightedness parameter (all other parameters are set to the known Kiudee-setting, we test here).


https://discordapp.com/channels/4254194 ... 8982059008
"Started a small tuning run for ShortSightedness. Same TC as my other tunes, with kiudee-bonus applied. Network: 58613.
Is there a plausible reason for why there is a bump in the middle?"

https://cdn.discordapp.com/attachments/ ... 26-339.png
Wow, nice. I think something like CLOP would work badly with so many parameters to tune and still having an acceptable time control in that many games. Imagine a fine tuned Lc0, as much fine tuned as Stockfish is.

User avatar
Laskos
Posts: 9843
Joined: Wed Jul 26, 2006 8:21 pm
Full name: Kai Laskos

Re: Lc0: Kiudee setting is very strong

Post by Laskos » Tue Jan 21, 2020 9:02 pm

pohl4711 wrote:
Tue Jan 21, 2020 9:40 am
Laskos wrote:
Tue Jan 21, 2020 9:23 am


Good! 1 + 2'' is already Blitz (large increment), so the optimum CPuct value is probably only smoothly increasing to LTC. Thanks for your test at 8' + 5'' in as many as 300 games, a huge task for a single machine. But it is needed, as these Kiudee parameters give a hard to believe boost. In the past I tried to fiddle with Lc0 engine parameters, it was always within even 1 SD error margins, it is hard to have many games at acceptable TC with Leela. Besides that, I do not understand what many of these parameters do precisely, and just blindly trying here and there is useless.
Kiudee settings are done by tuning-runs (Kiudee is the nickname of the author on discord). At the moment, he tunes the ShortSightedness parameter (all other parameters are set to the known Kiudee-setting, we test here).


https://discordapp.com/channels/4254194 ... 8982059008
"Started a small tuning run for ShortSightedness. Same TC as my other tunes, with kiudee-bonus applied. Network: 58613.
Is there a plausible reason for why there is a bump in the middle?"

https://cdn.discordapp.com/attachments/ ... 26-339.png

Against Stockfish 11 the same 40 Elo points difference. And I have never seen at any time in the past in any conditions Stockfish_dev beaten so badly by Leela (above 100 Elo points difference) on my computer. Time control is short Blitz. Normal games and scoring (non-Armageddon).

Code: Select all

RTX 2070 GPU
Fast 4 core i7 CPU
About ~2 Leela Ratio

TC: 2' + 1.2''

Score of SV_256x20_1541 vs SF_11: 36 - 18 - 46  [0.590] 100
Elo difference: 63.23 +/- 50.45
Finished match

Score of SV_256x20_1541_Kiudee vs SF_11: 42 - 13 - 45  [0.645] 100
Elo difference: 103.73 +/- 51.13
Finished match
+40 Elo points.

Remarkable results.

zullil
Posts: 5866
Joined: Mon Jan 08, 2007 11:31 pm
Location: PA USA
Full name: Louis Zulli

Re: Lc0: Kiudee setting is very strong

Post by zullil » Tue Jan 21, 2020 10:08 pm

Decided to try these settings, just for fun. Lc0 crushes the Fish!


Post Reply