Played 2 Gauntlets a 300 games (30''+300ms Bullet, with my 150 SALC Armageddon openings of my longtime-Testruns (https://www.sp-cc.de/nn-longtime-testing.htm)):
Lc0 default Leelenstein 13 Net vs. Lc0 Kiudee Leelenstein 13 Net and
Lc0 default Leelenstein 13 Net vs. Lc0 LSbinary Leelenstein 13 Net (the binary from josh patreon-site (post from 2019/12/26)
1 Lc0 0.23.2kiudee LS13 vs. Lc0 default LS13: 300 (+180,= 0,-120), 60.0 % (!!!)
2 Lc0 LSbinary LS13 vs. Lc0 default LS13: 300 (+103,= 0,-197), 34.3 %
Conclusions:
Josh-binary is very bad - do not use it!
Lc0 Kiudee is really impressive. 60%-40% means +70 Elo. But mention, that Armageddon (no draws, because all draws are counted as a win for Black) and Bullet-speed spread results, so +40 or +50 Elo seems more realistic. And on discord, some tests with other net-sizes (10x128 and T60) (Leelenstein Size is 20x256) show a measureable Elo gain with Kiudee-setting, too. So, it seems, that the Kiudee-setting should be the new default for Lc0. I will use it from now as default for my Lc0-testings.
Here the Kiudee setting:
CPuct=2.147
Fpu=0.443
PolicyTemperature=1.607
CPuctBase=18368
CPuctFactor=2.815
Lc0: Kiudee setting is very strong
Moderators: hgm, Rebel, chrisw
-
- Posts: 2434
- Joined: Sat Sep 03, 2011 7:25 am
- Location: Berlin, Germany
- Full name: Stefan Pohl
-
- Posts: 3657
- Joined: Wed Nov 18, 2015 11:41 am
- Location: hungary
Re: Lc0: Kiudee setting is very strong
Is there "Armageddon Elo" and a method to calculate "Armageddon Elo"?
I do not know about that.
Note
Kiudee-settings may be good for "Armageddon" but what is the situation in the case of normal games?
-
- Posts: 2434
- Joined: Sat Sep 03, 2011 7:25 am
- Location: Berlin, Germany
- Full name: Stefan Pohl
Re: Lc0: Kiudee setting is very strong
Armageddon means, all draws are counted as a win for Black. And the opening-positions give White an advantage (SALC: White can castle short, Black can castle long, only). Thats it. Because, there are no draws, the results are spreaded (more away from 50%-50%).
Here some tests from discord, with "normal" openings:
(both Nets are small (10x128). +32 and +53 Elo gain = +42 Elo average gain)
Gauntlet: J13B.2-188 vs lc0.net.62013 (default) / lc0.net.62013-tuned @kiudee
LC0-version: lc0-v0.23.2, Backend=cudnn-fp16
Hardware: RTX 2060
Software: Cutechess-CLI
Time control: 1k nodes/move
Book: openings-10ply-100k.pgn, 10 plies, sequential, color reversed
(Big T60 Net. +27 Elo gain)
Here some tests from discord, with "normal" openings:
Code: Select all
tc=1s+0.1s, RTX 2070
"bonus settings" cpuct=2.147, fpu=0.443, pst=1.607, cpuct-base=18368, cpuct-factor=2.815
# PLAYER : RATING ERROR POINTS PLAYED (%) CFS(%) W D L
1 lc0.net.58613.kiudee_bonus : 86 10 836.5 1422 58.8 100 459 755 208
2 lc0.net.58613.default : 33 9 699.0 1422 49.2 55 320 758 344
3 lc0.net.LD2.kiudee_bonus : 32 10 696.5 1422 49.0 100 308 777 337
4 lc0.net.LD2.default : 0 9 612.0 1422 43.0 --- 240 744 438
Gauntlet: J13B.2-188 vs lc0.net.62013 (default) / lc0.net.62013-tuned @kiudee
LC0-version: lc0-v0.23.2, Backend=cudnn-fp16
Hardware: RTX 2060
Software: Cutechess-CLI
Time control: 1k nodes/move
Book: openings-10ply-100k.pgn, 10 plies, sequential, color reversed
Code: Select all
# PLAYER : RATING ERROR POINTS PLAYED W L D D(%) CFS(%)
1 lc0.net.62013-tuned : 28.4 23.2 108.0 200 51 35 114 57 97
2 J13B.2-188 : 0.0 14.6 192.5 400 74 89 237 59 54
3 lc0.net.62013 : -1.8 24.4 99.5 200 38 39 123 62 ---
-
- Posts: 449
- Joined: Wed Nov 24, 2010 10:57 am
- Location: INDIA
Re: Lc0: Kiudee setting is very strong
pohl4711 wrote: ↑Mon Jan 20, 2020 2:38 pm Played 2 Gauntlets a 300 games (30''+300ms Bullet, with my 150 SALC Armageddon openings of my longtime-Testruns (https://www.sp-cc.de/nn-longtime-testing.htm)):
Lc0 default Leelenstein 13 Net vs. Lc0 Kiudee Leelenstein 13 Net and
Lc0 default Leelenstein 13 Net vs. Lc0 LSbinary Leelenstein 13 Net (the binary from josh patreon-site (post from 2019/12/26)
1 Lc0 0.23.2kiudee LS13 vs. Lc0 default LS13: 300 (+180,= 0,-120), 60.0 % (!!!)
2 Lc0 LSbinary LS13 vs. Lc0 default LS13: 300 (+103,= 0,-197), 34.3 %
Conclusions:
Josh-binary is very bad - do not use it!
Lc0 Kiudee is really impressive. 60%-40% means +70 Elo. But mention, that Armageddon (no draws, because all draws are counted as a win for Black) and Bullet-speed spread results, so +40 or +50 Elo seems more realistic. And on discord, some tests with other net-sizes (10x128 and T60) (Leelenstein Size is 20x256) show a measureable Elo gain with Kiudee-setting, too. So, it seems, that the Kiudee-setting should be the new default for Lc0. I will use it from now as default for my Lc0-testings.
Here the Kiudee setting:
CPuct=2.147
Fpu=0.443
PolicyTemperature=1.607
CPuctBase=18368
CPuctFactor=2.815
Whats Kiudee settings ?
Edit: I got it
Last edited by Master Om on Mon Jan 20, 2020 4:28 pm, edited 1 time in total.
Always Expect the Unexpected
-
- Posts: 157
- Joined: Tue Apr 30, 2013 1:29 am
Re: Lc0: Kiudee setting is very strong
Any idea if it still holds for very long TCs or does it converge to the same thing given enough time (which would be my intuition).
-
- Posts: 2727
- Joined: Wed May 12, 2010 10:00 pm
Re: Lc0: Kiudee setting is very strong
Thanks, I will give them a try. And I agree that the default settings. Are not the best by a long shot. I never run default with Lc0 in my testing. I will see if this is better. If it is better then my settings. I will use them in my testing.pohl4711 wrote: ↑Mon Jan 20, 2020 2:38 pm Played 2 Gauntlets a 300 games (30''+300ms Bullet, with my 150 SALC Armageddon openings of my longtime-Testruns (https://www.sp-cc.de/nn-longtime-testing.htm)):
Lc0 default Leelenstein 13 Net vs. Lc0 Kiudee Leelenstein 13 Net and
Lc0 default Leelenstein 13 Net vs. Lc0 LSbinary Leelenstein 13 Net (the binary from josh patreon-site (post from 2019/12/26)
1 Lc0 0.23.2kiudee LS13 vs. Lc0 default LS13: 300 (+180,= 0,-120), 60.0 % (!!!)
2 Lc0 LSbinary LS13 vs. Lc0 default LS13: 300 (+103,= 0,-197), 34.3 %
Conclusions:
Josh-binary is very bad - do not use it!
Lc0 Kiudee is really impressive. 60%-40% means +70 Elo. But mention, that Armageddon (no draws, because all draws are counted as a win for Black) and Bullet-speed spread results, so +40 or +50 Elo seems more realistic. And on discord, some tests with other net-sizes (10x128 and T60) (Leelenstein Size is 20x256) show a measureable Elo gain with Kiudee-setting, too. So, it seems, that the Kiudee-setting should be the new default for Lc0. I will use it from now as default for my Lc0-testings.
Here the Kiudee setting:
CPuct=2.147
Fpu=0.443
PolicyTemperature=1.607
CPuctBase=18368
CPuctFactor=2.815
"The worst thing that can happen to a forum is a running wild attacking moderator(HGM) who is not corrected by the community." - Ed Schröder
But my words like silent raindrops fell. And echoed in the wells of silence.
But my words like silent raindrops fell. And echoed in the wells of silence.
-
- Posts: 2727
- Joined: Wed May 12, 2010 10:00 pm
Re: Lc0: Kiudee setting is very strong
We will have to try the setting and find out. But if it does get you to the same place, but quicker IMO it would still be better then default.OneTrickPony wrote: ↑Mon Jan 20, 2020 4:19 pm Any idea if it still holds for very long TCs or does it converge to the same thing given enough time (which would be my intuition).
Neural Nets are so much different with the search. All roll outs play to the end of the game. So you always have 100 percent depth. What you are selecting is the width of the search, and what moves you keep and throw away at each "ply" or roll out.
"The worst thing that can happen to a forum is a running wild attacking moderator(HGM) who is not corrected by the community." - Ed Schröder
But my words like silent raindrops fell. And echoed in the wells of silence.
But my words like silent raindrops fell. And echoed in the wells of silence.
-
- Posts: 10948
- Joined: Wed Jul 26, 2006 10:21 pm
- Full name: Kai Laskos
Re: Lc0: Kiudee setting is very strong
Thanks for the info, I can confirm at 30'' + 0.3'' in 200 games from unbalanced openings:pohl4711 wrote: ↑Mon Jan 20, 2020 2:38 pm Played 2 Gauntlets a 300 games (30''+300ms Bullet, with my 150 SALC Armageddon openings of my longtime-Testruns (https://www.sp-cc.de/nn-longtime-testing.htm)):
Lc0 default Leelenstein 13 Net vs. Lc0 Kiudee Leelenstein 13 Net and
Lc0 default Leelenstein 13 Net vs. Lc0 LSbinary Leelenstein 13 Net (the binary from josh patreon-site (post from 2019/12/26)
1 Lc0 0.23.2kiudee LS13 vs. Lc0 default LS13: 300 (+180,= 0,-120), 60.0 % (!!!)
2 Lc0 LSbinary LS13 vs. Lc0 default LS13: 300 (+103,= 0,-197), 34.3 %
Conclusions:
Josh-binary is very bad - do not use it!
Lc0 Kiudee is really impressive. 60%-40% means +70 Elo. But mention, that Armageddon (no draws, because all draws are counted as a win for Black) and Bullet-speed spread results, so +40 or +50 Elo seems more realistic. And on discord, some tests with other net-sizes (10x128 and T60) (Leelenstein Size is 20x256) show a measureable Elo gain with Kiudee-setting, too. So, it seems, that the Kiudee-setting should be the new default for Lc0. I will use it from now as default for my Lc0-testings.
Here the Kiudee setting:
CPuct=2.147
Fpu=0.443
PolicyTemperature=1.607
CPuctBase=18368
CPuctFactor=2.815
Code: Select all
Score of SV_256x20_1541_Kiudee vs SV_256x20_1541: 52 - 30 - 118 [0.555] 200
Elo difference: 38.37 +/- 30.78
Finished match
With unbalanced openings in paired games one should use the pentanomial error margins, which are smaller than the usual trinomial ones. Here is how a result looks in my case:
So, all in all about 40 +/- 20 (95% confidence) Elo points improvement, which is amazing. I guess the optimal CPuct will increase with longer time control, but I have no resources to tune it to various TC.
Again, 40 Elo points gain is amazing.
-
- Posts: 3657
- Joined: Wed Nov 18, 2015 11:41 am
- Location: hungary
Re: Lc0: Kiudee setting is very strong
Is our reported result an "Armageddon" result or a normal, really experienced result?Laskos wrote: ↑Tue Jan 21, 2020 12:22 am ...
Thanks for the info, I can confirm at 30'' + 0.3'' in 200 games from unbalanced openings:The engine is Lc0 v0.23.2.Code: Select all
Score of SV_256x20_1541_Kiudee vs SV_256x20_1541: 52 - 30 - 118 [0.555] 200 Elo difference: 38.37 +/- 30.78 Finished match
With unbalanced openings in paired games one should use the pentanomial error margins, which are smaller than the usual trinomial ones. Here is how a result looks in my case:
So, all in all about 40 +/- 20 (95% confidence) Elo points improvement, which is amazing. I guess the optimal CPuct will increase with longer time control, but I have no resources to tune it to various TC.
Again, 40 Elo points gain is amazing.
Using unbalanced start positions is not equivalent to using "Armageddon" method to modify the result of games.
-
- Posts: 10948
- Joined: Wed Jul 26, 2006 10:21 pm
- Full name: Kai Laskos
Re: Lc0: Kiudee setting is very strong
No, regular 3-mover human openings, somewhat unbalanced to avoid 90% draw rates. From games of above 2200 FIDE Elo points humans.corres wrote: ↑Tue Jan 21, 2020 12:47 amIs our reported result an "Armageddon" result or a normal, really experienced result?Laskos wrote: ↑Tue Jan 21, 2020 12:22 am ...
Thanks for the info, I can confirm at 30'' + 0.3'' in 200 games from unbalanced openings:The engine is Lc0 v0.23.2.Code: Select all
Score of SV_256x20_1541_Kiudee vs SV_256x20_1541: 52 - 30 - 118 [0.555] 200 Elo difference: 38.37 +/- 30.78 Finished match
With unbalanced openings in paired games one should use the pentanomial error margins, which are smaller than the usual trinomial ones. Here is how a result looks in my case:
So, all in all about 40 +/- 20 (95% confidence) Elo points improvement, which is amazing. I guess the optimal CPuct will increase with longer time control, but I have no resources to tune it to various TC.
Again, 40 Elo points gain is amazing.
Using unbalanced start positions is not equivalent to using "Armageddon" method to modify the result of games.