Lc0: Kiudee setting is very strong

Discussion of anything and everything relating to chess playing software and machines.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
pohl4711
Posts: 1162
Joined: Sat Sep 03, 2011 5:25 am
Location: Berlin, Germany
Contact:

Lc0: Kiudee setting is very strong

Post by pohl4711 » Mon Jan 20, 2020 1:38 pm

Played 2 Gauntlets a 300 games (30''+300ms Bullet, with my 150 SALC Armageddon openings of my longtime-Testruns (https://www.sp-cc.de/nn-longtime-testing.htm)):
Lc0 default Leelenstein 13 Net vs. Lc0 Kiudee Leelenstein 13 Net and
Lc0 default Leelenstein 13 Net vs. Lc0 LSbinary Leelenstein 13 Net (the binary from josh patreon-site (post from 2019/12/26)

1 Lc0 0.23.2kiudee LS13 vs. Lc0 default LS13: 300 (+180,= 0,-120), 60.0 % (!!!)

2 Lc0 LSbinary LS13 vs. Lc0 default LS13: 300 (+103,= 0,-197), 34.3 %

Conclusions:
Josh-binary is very bad - do not use it!
Lc0 Kiudee is really impressive. 60%-40% means +70 Elo. But mention, that Armageddon (no draws, because all draws are counted as a win for Black) and Bullet-speed spread results, so +40 or +50 Elo seems more realistic. And on discord, some tests with other net-sizes (10x128 and T60) (Leelenstein Size is 20x256) show a measureable Elo gain with Kiudee-setting, too. So, it seems, that the Kiudee-setting should be the new default for Lc0. I will use it from now as default for my Lc0-testings.

Here the Kiudee setting:

CPuct=2.147
Fpu=0.443
PolicyTemperature=1.607
CPuctBase=18368
CPuctFactor=2.815

corres
Posts: 2026
Joined: Wed Nov 18, 2015 10:41 am
Location: hungary

Re: Lc0: Kiudee setting is very strong

Post by corres » Mon Jan 20, 2020 2:51 pm

pohl4711 wrote:
Mon Jan 20, 2020 1:38 pm
...
Lc0 Kiudee is really impressive. 60%-40% means +70 Elo.
But mention, that Armageddon (no draws, because all draws are counted as a win for Black) and Bullet-speed spread results, so +40 or +50 Elo seems more realistic.
...
Is there "Armageddon Elo" and a method to calculate "Armageddon Elo"?
I do not know about that.
Note
Kiudee-settings may be good for "Armageddon" but what is the situation in the case of normal games?

pohl4711
Posts: 1162
Joined: Sat Sep 03, 2011 5:25 am
Location: Berlin, Germany
Contact:

Re: Lc0: Kiudee setting is very strong

Post by pohl4711 » Mon Jan 20, 2020 3:14 pm

Armageddon means, all draws are counted as a win for Black. And the opening-positions give White an advantage (SALC: White can castle short, Black can castle long, only). Thats it. Because, there are no draws, the results are spreaded (more away from 50%-50%).

Here some tests from discord, with "normal" openings:

Code: Select all

tc=1s+0.1s, RTX 2070
"bonus settings" cpuct=2.147, fpu=0.443, pst=1.607, cpuct-base=18368, cpuct-factor=2.815

   # PLAYER                        :  RATING  ERROR  POINTS  PLAYED   (%)  CFS(%)    W    D    L
   1 lc0.net.58613.kiudee_bonus    :      86     10   836.5    1422  58.8     100  459  755  208
   2 lc0.net.58613.default         :      33      9   699.0    1422  49.2      55  320  758  344
   3 lc0.net.LD2.kiudee_bonus      :      32     10   696.5    1422  49.0     100  308  777  337
   4 lc0.net.LD2.default           :       0      9   612.0    1422  43.0     ---  240  744  438
(both Nets are small (10x128). +32 and +53 Elo gain = +42 Elo average gain)


Gauntlet: J13B.2-188 vs lc0.net.62013 (default) / lc0.net.62013-tuned @kiudee
LC0-version: lc0-v0.23.2, Backend=cudnn-fp16
Hardware: RTX 2060
Software: Cutechess-CLI
Time control: 1k nodes/move
Book: openings-10ply-100k.pgn, 10 plies, sequential, color reversed

Code: Select all

# PLAYER                 :  RATING  ERROR  POINTS  PLAYED    W    L    D  D(%)  CFS(%)
1 lc0.net.62013-tuned    :    28.4   23.2   108.0     200   51   35  114    57      97
2 J13B.2-188             :     0.0   14.6   192.5     400   74   89  237    59      54
3 lc0.net.62013          :    -1.8   24.4    99.5     200   38   39  123    62     ---
(Big T60 Net. +27 Elo gain)

User avatar
Master Om
Posts: 348
Joined: Wed Nov 24, 2010 9:57 am
Location: INDIA

Re: Lc0: Kiudee setting is very strong

Post by Master Om » Mon Jan 20, 2020 3:17 pm

pohl4711 wrote:
Mon Jan 20, 2020 1:38 pm
Played 2 Gauntlets a 300 games (30''+300ms Bullet, with my 150 SALC Armageddon openings of my longtime-Testruns (https://www.sp-cc.de/nn-longtime-testing.htm)):
Lc0 default Leelenstein 13 Net vs. Lc0 Kiudee Leelenstein 13 Net and
Lc0 default Leelenstein 13 Net vs. Lc0 LSbinary Leelenstein 13 Net (the binary from josh patreon-site (post from 2019/12/26)

1 Lc0 0.23.2kiudee LS13 vs. Lc0 default LS13: 300 (+180,= 0,-120), 60.0 % (!!!)

2 Lc0 LSbinary LS13 vs. Lc0 default LS13: 300 (+103,= 0,-197), 34.3 %

Conclusions:
Josh-binary is very bad - do not use it!
Lc0 Kiudee is really impressive. 60%-40% means +70 Elo. But mention, that Armageddon (no draws, because all draws are counted as a win for Black) and Bullet-speed spread results, so +40 or +50 Elo seems more realistic. And on discord, some tests with other net-sizes (10x128 and T60) (Leelenstein Size is 20x256) show a measureable Elo gain with Kiudee-setting, too. So, it seems, that the Kiudee-setting should be the new default for Lc0. I will use it from now as default for my Lc0-testings.

Here the Kiudee setting:

CPuct=2.147
Fpu=0.443
PolicyTemperature=1.607
CPuctBase=18368
CPuctFactor=2.815

Whats Kiudee settings ?
Edit: I got it
Last edited by Master Om on Mon Jan 20, 2020 3:28 pm, edited 1 time in total.
Always Expect the Unexpected

OneTrickPony
Posts: 119
Joined: Mon Apr 29, 2013 11:29 pm

Re: Lc0: Kiudee setting is very strong

Post by OneTrickPony » Mon Jan 20, 2020 3:19 pm

Any idea if it still holds for very long TCs or does it converge to the same thing given enough time (which would be my intuition).

mwyoung
Posts: 1937
Joined: Wed May 12, 2010 8:00 pm

Re: Lc0: Kiudee setting is very strong

Post by mwyoung » Mon Jan 20, 2020 7:05 pm

pohl4711 wrote:
Mon Jan 20, 2020 1:38 pm
Played 2 Gauntlets a 300 games (30''+300ms Bullet, with my 150 SALC Armageddon openings of my longtime-Testruns (https://www.sp-cc.de/nn-longtime-testing.htm)):
Lc0 default Leelenstein 13 Net vs. Lc0 Kiudee Leelenstein 13 Net and
Lc0 default Leelenstein 13 Net vs. Lc0 LSbinary Leelenstein 13 Net (the binary from josh patreon-site (post from 2019/12/26)

1 Lc0 0.23.2kiudee LS13 vs. Lc0 default LS13: 300 (+180,= 0,-120), 60.0 % (!!!)

2 Lc0 LSbinary LS13 vs. Lc0 default LS13: 300 (+103,= 0,-197), 34.3 %

Conclusions:
Josh-binary is very bad - do not use it!
Lc0 Kiudee is really impressive. 60%-40% means +70 Elo. But mention, that Armageddon (no draws, because all draws are counted as a win for Black) and Bullet-speed spread results, so +40 or +50 Elo seems more realistic. And on discord, some tests with other net-sizes (10x128 and T60) (Leelenstein Size is 20x256) show a measureable Elo gain with Kiudee-setting, too. So, it seems, that the Kiudee-setting should be the new default for Lc0. I will use it from now as default for my Lc0-testings.

Here the Kiudee setting:

CPuct=2.147
Fpu=0.443
PolicyTemperature=1.607
CPuctBase=18368
CPuctFactor=2.815
Thanks, I will give them a try. And I agree that the default settings. Are not the best by a long shot. I never run default with Lc0 in my testing. I will see if this is better. If it is better then my settings. I will use them in my testing.
Professing themselves to be wise, they became fools,
take on me. Foes 0.

mwyoung
Posts: 1937
Joined: Wed May 12, 2010 8:00 pm

Re: Lc0: Kiudee setting is very strong

Post by mwyoung » Mon Jan 20, 2020 7:09 pm

OneTrickPony wrote:
Mon Jan 20, 2020 3:19 pm
Any idea if it still holds for very long TCs or does it converge to the same thing given enough time (which would be my intuition).
We will have to try the setting and find out. But if it does get you to the same place, but quicker IMO it would still be better then default.

Neural Nets are so much different with the search. All roll outs play to the end of the game. So you always have 100 percent depth. What you are selecting is the width of the search, and what moves you keep and throw away at each "ply" or roll out.
Professing themselves to be wise, they became fools,
take on me. Foes 0.

User avatar
Laskos
Posts: 9840
Joined: Wed Jul 26, 2006 8:21 pm
Full name: Kai Laskos

Re: Lc0: Kiudee setting is very strong

Post by Laskos » Mon Jan 20, 2020 11:22 pm

pohl4711 wrote:
Mon Jan 20, 2020 1:38 pm
Played 2 Gauntlets a 300 games (30''+300ms Bullet, with my 150 SALC Armageddon openings of my longtime-Testruns (https://www.sp-cc.de/nn-longtime-testing.htm)):
Lc0 default Leelenstein 13 Net vs. Lc0 Kiudee Leelenstein 13 Net and
Lc0 default Leelenstein 13 Net vs. Lc0 LSbinary Leelenstein 13 Net (the binary from josh patreon-site (post from 2019/12/26)

1 Lc0 0.23.2kiudee LS13 vs. Lc0 default LS13: 300 (+180,= 0,-120), 60.0 % (!!!)

2 Lc0 LSbinary LS13 vs. Lc0 default LS13: 300 (+103,= 0,-197), 34.3 %

Conclusions:
Josh-binary is very bad - do not use it!
Lc0 Kiudee is really impressive. 60%-40% means +70 Elo. But mention, that Armageddon (no draws, because all draws are counted as a win for Black) and Bullet-speed spread results, so +40 or +50 Elo seems more realistic. And on discord, some tests with other net-sizes (10x128 and T60) (Leelenstein Size is 20x256) show a measureable Elo gain with Kiudee-setting, too. So, it seems, that the Kiudee-setting should be the new default for Lc0. I will use it from now as default for my Lc0-testings.

Here the Kiudee setting:

CPuct=2.147
Fpu=0.443
PolicyTemperature=1.607
CPuctBase=18368
CPuctFactor=2.815
Thanks for the info, I can confirm at 30'' + 0.3'' in 200 games from unbalanced openings:

Code: Select all

Score of SV_256x20_1541_Kiudee vs SV_256x20_1541: 52 - 30 - 118  [0.555] 200
Elo difference: 38.37 +/- 30.78
Finished match
The engine is Lc0 v0.23.2.

With unbalanced openings in paired games one should use the pentanomial error margins, which are smaller than the usual trinomial ones. Here is how a result looks in my case:

Image

So, all in all about 40 +/- 20 (95% confidence) Elo points improvement, which is amazing. I guess the optimal CPuct will increase with longer time control, but I have no resources to tune it to various TC.

Again, 40 Elo points gain is amazing.

corres
Posts: 2026
Joined: Wed Nov 18, 2015 10:41 am
Location: hungary

Re: Lc0: Kiudee setting is very strong

Post by corres » Mon Jan 20, 2020 11:47 pm

Laskos wrote:
Mon Jan 20, 2020 11:22 pm
...
Thanks for the info, I can confirm at 30'' + 0.3'' in 200 games from unbalanced openings:

Code: Select all

Score of SV_256x20_1541_Kiudee vs SV_256x20_1541: 52 - 30 - 118  [0.555] 200
Elo difference: 38.37 +/- 30.78
Finished match
The engine is Lc0 v0.23.2.
With unbalanced openings in paired games one should use the pentanomial error margins, which are smaller than the usual trinomial ones. Here is how a result looks in my case:
Image
So, all in all about 40 +/- 20 (95% confidence) Elo points improvement, which is amazing. I guess the optimal CPuct will increase with longer time control, but I have no resources to tune it to various TC.
Again, 40 Elo points gain is amazing.
Is our reported result an "Armageddon" result or a normal, really experienced result?
Using unbalanced start positions is not equivalent to using "Armageddon" method to modify the result of games.

User avatar
Laskos
Posts: 9840
Joined: Wed Jul 26, 2006 8:21 pm
Full name: Kai Laskos

Re: Lc0: Kiudee setting is very strong

Post by Laskos » Mon Jan 20, 2020 11:49 pm

corres wrote:
Mon Jan 20, 2020 11:47 pm
Laskos wrote:
Mon Jan 20, 2020 11:22 pm
...
Thanks for the info, I can confirm at 30'' + 0.3'' in 200 games from unbalanced openings:

Code: Select all

Score of SV_256x20_1541_Kiudee vs SV_256x20_1541: 52 - 30 - 118  [0.555] 200
Elo difference: 38.37 +/- 30.78
Finished match
The engine is Lc0 v0.23.2.
With unbalanced openings in paired games one should use the pentanomial error margins, which are smaller than the usual trinomial ones. Here is how a result looks in my case:
Image
So, all in all about 40 +/- 20 (95% confidence) Elo points improvement, which is amazing. I guess the optimal CPuct will increase with longer time control, but I have no resources to tune it to various TC.
Again, 40 Elo points gain is amazing.
Is our reported result an "Armageddon" result or a normal, really experienced result?
Using unbalanced start positions is not equivalent to using "Armageddon" method to modify the result of games.
No, regular 3-mover human openings, somewhat unbalanced to avoid 90% draw rates. From games of above 2200 FIDE Elo points humans.

Post Reply