Lc0: Kiudee setting is very strong

[corres]

...
Thanks for the info, I can confirm at 30'' + 0.3'' in 200 games from unbalanced openings:

Code: Select all

Score of SV_256x20_1541_Kiudee vs SV_256x20_1541: 52 - 30 - 118  [0.555] 200
Elo difference: 38.37 +/- 30.78
Finished match

The engine is Lc0 v0.23.2.
With unbalanced openings in paired games one should use the pentanomial error margins, which are smaller than the usual trinomial ones. Here is how a result looks in my case:

So, all in all about 40 +/- 20 (95% confidence) Elo points improvement, which is amazing. I guess the optimal CPuct will increase with longer time control, but I have no resources to tune it to various TC.
Again, 40 Elo points gain is amazing.

Is our reported result an "Armageddon" result or a normal, really experienced result?
Using unbalanced start positions is not equivalent to using "Armageddon" method to modify the result of games.

No, regular 3-mover human openings, somewhat unbalanced to avoid 90% draw rates. From games of above 2200 FIDE Elo points humans.

Without "Armageddon"?

[Laskos]

...
Thanks for the info, I can confirm at 30'' + 0.3'' in 200 games from unbalanced openings:

Code: Select all

Score of SV_256x20_1541_Kiudee vs SV_256x20_1541: 52 - 30 - 118  [0.555] 200
Elo difference: 38.37 +/- 30.78
Finished match

The engine is Lc0 v0.23.2.
With unbalanced openings in paired games one should use the pentanomial error margins, which are smaller than the usual trinomial ones. Here is how a result looks in my case:

So, all in all about 40 +/- 20 (95% confidence) Elo points improvement, which is amazing. I guess the optimal CPuct will increase with longer time control, but I have no resources to tune it to various TC.
Again, 40 Elo points gain is amazing.

Is our reported result an "Armageddon" result or a normal, really experienced result?
Using unbalanced start positions is not equivalent to using "Armageddon" method to modify the result of games.

No, regular 3-mover human openings, somewhat unbalanced to avoid 90% draw rates. From games of above 2200 FIDE Elo points humans.

Without "Armageddon"?

No, no, normal games and scoring. Armageddon is a bit different chess, I am not testing with it as of yet. Unbalanced openings are almost as good as resolving power goes, but it's actual chess and scoring.

[corres]

...
Again, 40 Elo points gain is amazing.

Is our reported result an "Armageddon" result or a normal, really experienced result?
Using unbalanced start positions is not equivalent to using "Armageddon" method to modify the result of games.

No, regular 3-mover human openings, somewhat unbalanced to avoid 90% draw rates. From games of above 2200 FIDE Elo points humans.

Without "Armageddon"?

No, no, normal games and scoring. Armageddon is a bit different chess, I am not testing with it as of yet. Unbalanced openings are almost as good as resolving power goes, but it's actual chess and scoring.

Thanks for the answers.

[bmp1974]

Has anyone observed improvement with regard to endgame play by Lc0? Does the Kiudee settings avoid Lc0 trolling in endgames?
My prediction is that, in coming few months, with T60 NN reaching elo 3000 mark and with settings that help improve endgame play quality, Lc0 will be unbeatable!

[pohl4711]

Thanks for the info, I can confirm at 30'' + 0.3'' in 200 games from unbalanced openings:

Code: Select all

Score of SV_256x20_1541_Kiudee vs SV_256x20_1541: 52 - 30 - 118  [0.555] 200
Elo difference: 38.37 +/- 30.78
Finished match

On Twitch (potato_chess), 2 testruns with Net 62038 were played (1'+2'' on RTX, which is not that short and 62038 is a bigger Net (24x320)). 200 games each. And with Kiudee, the result was +36 Elo better, than the default setting. So, in each testrun, I have seen, Kiudee scored at least +30 Elo better. Nice!



I will repeat my longtime-testrun (8'+5'') of Lc0 0.23.1 LS 12.2 with Kiudee-setting. Lets see, if there is an Elo-gain with long thinkingtime, too...but this will take 6 days from now.

https://www.sp-cc.de/nn-longtime-testing.htm

[Laskos]

Thanks for the info, I can confirm at 30'' + 0.3'' in 200 games from unbalanced openings:

Code: Select all

Score of SV_256x20_1541_Kiudee vs SV_256x20_1541: 52 - 30 - 118  [0.555] 200
Elo difference: 38.37 +/- 30.78
Finished match

On Twitch (potato_chess), 2 testruns with Net 62038 were played (1'+2'' on RTX, which is not that short and 62038 is a bigger Net (24x320)). 200 games each. And with Kiudee, the result was +36 Elo better, than the default setting. So, in each testrun, I have seen, Kiudee scored at least +30 Elo better. Nice!



I will repeat my longtime-testrun (8'+5'') of Lc0 0.23.1 LS 12.2 with Kiudee-setting. Lets see, if there is an Elo-gain with long thinkingtime, too...but this will take 6 days from now.

https://www.sp-cc.de/nn-longtime-testing.htm

Good! 1 + 2'' is already Blitz (large increment), so the optimum CPuct value is probably only smoothly increasing to LTC. Thanks for your test at 8' + 5'' in as many as 300 games, a huge task for a single machine. But it is needed, as these Kiudee parameters give a hard to believe boost. In the past I tried to fiddle with Lc0 engine parameters, it was always within even 1 SD error margins, it is hard to have many games at acceptable TC with Leela. Besides that, I do not understand what many of these parameters do precisely, and just blindly trying here and there is useless.

Thanks for posting them, now I will modify Cutechess json file for Lc0 0232 with different nets, all to take these parameters.

[pohl4711]

Good! 1 + 2'' is already Blitz (large increment), so the optimum CPuct value is probably only smoothly increasing to LTC. Thanks for your test at 8' + 5'' in as many as 300 games, a huge task for a single machine. But it is needed, as these Kiudee parameters give a hard to believe boost. In the past I tried to fiddle with Lc0 engine parameters, it was always within even 1 SD error margins, it is hard to have many games at acceptable TC with Leela. Besides that, I do not understand what many of these parameters do precisely, and just blindly trying here and there is useless.

Kiudee settings are done by tuning-runs (Kiudee is the nickname of the author on discord). At the moment, he tunes the ShortSightedness parameter (all other parameters are set to the known Kiudee-setting, we test here).


https://discordapp.com/channels/4254194 ... 8982059008
"Started a small tuning run for ShortSightedness. Same TC as my other tunes, with kiudee-bonus applied. Network: 58613.
Is there a plausible reason for why there is a bump in the middle?"

https://cdn.discordapp.com/attachments/ ... 26-339.png

[Laskos]

Good! 1 + 2'' is already Blitz (large increment), so the optimum CPuct value is probably only smoothly increasing to LTC. Thanks for your test at 8' + 5'' in as many as 300 games, a huge task for a single machine. But it is needed, as these Kiudee parameters give a hard to believe boost. In the past I tried to fiddle with Lc0 engine parameters, it was always within even 1 SD error margins, it is hard to have many games at acceptable TC with Leela. Besides that, I do not understand what many of these parameters do precisely, and just blindly trying here and there is useless.

Kiudee settings are done by tuning-runs (Kiudee is the nickname of the author on discord). At the moment, he tunes the ShortSightedness parameter (all other parameters are set to the known Kiudee-setting, we test here).


https://discordapp.com/channels/4254194 ... 8982059008
"Started a small tuning run for ShortSightedness. Same TC as my other tunes, with kiudee-bonus applied. Network: 58613.
Is there a plausible reason for why there is a bump in the middle?"

https://cdn.discordapp.com/attachments/ ... 26-339.png

Wow, nice. I think something like CLOP would work badly with so many parameters to tune and still having an acceptable time control in that many games. Imagine a fine tuned Lc0, as much fine tuned as Stockfish is.

[Laskos]

Good! 1 + 2'' is already Blitz (large increment), so the optimum CPuct value is probably only smoothly increasing to LTC. Thanks for your test at 8' + 5'' in as many as 300 games, a huge task for a single machine. But it is needed, as these Kiudee parameters give a hard to believe boost. In the past I tried to fiddle with Lc0 engine parameters, it was always within even 1 SD error margins, it is hard to have many games at acceptable TC with Leela. Besides that, I do not understand what many of these parameters do precisely, and just blindly trying here and there is useless.

Kiudee settings are done by tuning-runs (Kiudee is the nickname of the author on discord). At the moment, he tunes the ShortSightedness parameter (all other parameters are set to the known Kiudee-setting, we test here).


https://discordapp.com/channels/4254194 ... 8982059008
"Started a small tuning run for ShortSightedness. Same TC as my other tunes, with kiudee-bonus applied. Network: 58613.
Is there a plausible reason for why there is a bump in the middle?"

https://cdn.discordapp.com/attachments/ ... 26-339.png

Against Stockfish 11 the same 40 Elo points difference. And I have never seen at any time in the past in any conditions Stockfish_dev beaten so badly by Leela (above 100 Elo points difference) on my computer. Time control is short Blitz. Normal games and scoring (non-Armageddon).

Code: Select all

RTX 2070 GPU
Fast 4 core i7 CPU
About ~2 Leela Ratio

TC: 2' + 1.2''

Score of SV_256x20_1541 vs SF_11: 36 - 18 - 46  [0.590] 100
Elo difference: 63.23 +/- 50.45
Finished match

Score of SV_256x20_1541_Kiudee vs SF_11: 42 - 13 - 45  [0.645] 100
Elo difference: 103.73 +/- 51.13
Finished match

+40 Elo points.

Remarkable results.

[zullil]

Decided to try these settings, just for fun. Lc0 crushes the Fish!

[pgn][White "Lc0 (384x30-t40-swa-1705000)"]
[Black "Stockfish 11 (20 threads)"]
[Result "1-0"]
[TimeControl "180s+3s"]

1. e4 c5 2. Nf3 d6 3. d4 cxd4 4. Nxd4 Nf6 5. Nc3 a6 6. Be3 e5 7. Nb3 Be7 8. f3
Be6 9. Qd2 O-O 10. O-O-O Nbd7 11. g4 b5 12. g5 Nh5 13. Nd5 Bxd5 14. exd5 f5 15.
Rg1 f4 16. Bf2 Bxg5 17. Bh3 Be7 18. Na5 Kh8 19. Nc6 Qe8 20. Kb1 Nhf6 21. Be6
Bd8 22. Qb4 Bc7 23. a4 Qh5 24. Rd3 Qxh2 25. Qe1 Rae8 26. Qf1 e4 27. Rh1 Qxh1
28. Qxh1 exd3 29. cxd3 Nc5 30. Bxc5 dxc5 31. Qg1 Bd6 32. a5 b4 33. d4 cxd4 34.
Ka2 Ra8 35. Qxd4 Ne8 36. Qc4 Nc7 37. Bd7 Rf7 38. Be6 Rf6 39. Nd4 Be5 40. Kb3
Rh6 41. Bf5 Bxd4 42. Qxc7 Bf6 43. Qf7 Be5 44. Qe7 Bf6 45. Qb7 Rd8 46. Qxa6 Rh2
47. Bc2 Rd2 48. Qe6 R2xd5 49. a6 Bd4 50. Qe7 R5d7 51. Qh4 h6 52. Qxf4 Re7 53.
Qf5 Kg8 54. Be4 Bf2 55. Kxb4 Red7 56. Ka5 Rd6 57. b4 g6 58. Qe5 Bb6+ 59. Kb5
Bd4 60. Qe7 R8d7 61. Qe8+ Kg7 62. Bxg6 Ra7 63. Be4 Rdxa6 64. Bd5 Ba1 65. f4 Kf6
66. Qf8+ Kg6 67. Be4+ Kh5 68. Bf5 Kh4 69. Qc5 Kg3 70. Qe3+ Kg2 71. Bh3+ Kh2 72.
Bf1 Ra2 73. Qh3+ Kg1 74. Bc4 Bf6 75. Bxa2 Rb7+ 76. Kc6 Rg7 77. b5 Bd4 78. b6
Rg6+ 79. Kd5 Bf2 80. b7 Ba7 81. f5 Rg2 82. Qxh6 Rxa2 83. Qb6+ Bxb6 84. b8=Q
Rd2+ 85. Kc6 Bd8 86. f6 Bxf6 87. Qb1+ Kf2 88. Qf5+ Ke3 89. Qxf6 Rd4 90. Qg6 Ra4
91. Qg3+ Ke2 92. Qc3 Rg4 93. Kd5 Rh4 94. Qc2+ Kf3 95. Qd3+ Kf2 96. Ke5 Rh8 97.
Qa3 Rh5+ 98. Kf4 Rh4+ 99. Kf5 Rh7 100. Qb2+ Ke3 101. Qe5+ Kf3 102. Qd5+ Ke3
103. Kf6 Rh8 104. Kg5 Re8 105. Qc5+ Kd2 106. Qa5+ Kc1 107. Qc7+ Kb1 108. Qb7+
Ka1 109. Qa6+ Kb1 110. Qb5+ Ka1 111. Qxe8 Kb2 112. Kf4 Kc3 113. Ke4 Kb4 114.
Kd4 Kb3 115. Qe2 Kb4 116. Qb2+ Ka5 117. Kc5 Ka4 118. Qb4# 1-0[/pgn]