New CLOP settings give Leela huge tactics boost

[Albert Silver]

In view of the many settings in LC0, I ran a deep CLOP on three big ones at the same time. CLOP is designed to tune for the best playing performance, not best tactics or anything. The old default settings were nearly catastrophic giving NN369 a mere 109/200 solved in my corrected WAC set. The new settings yield... 159/200.



You can try them without any work, by downloading the latest LC0 June 4 build at https://crem.xyz/lc0/ dubbed experimental.

[Milos]

In view of the many settings in LC0, I ran a deep CLOP on three big ones at the same time. CLOP is designed to tune for the best playing performance, not best tactics or anything. The old default settings were nearly catastrophic giving NN369 a mere 109/200 solved in my corrected WAC set. The new settings yield... 159/200.



You can try them without any work, by downloading the latest LC0 June 4 build at https://crem.xyz/lc0/ dubbed experimental.

Instead of running meaningless "tuning" try at least beating the default version with your "magnificent" tuned parameters.

[carldaman]

In view of the many settings in LC0, I ran a deep CLOP on three big ones at the same time. CLOP is designed to tune for the best playing performance, not best tactics or anything. The old default settings were nearly catastrophic giving NN369 a mere 109/200 solved in my corrected WAC set. The new settings yield... 159/200.



You can try them without any work, by downloading the latest LC0 June 4 build at https://crem.xyz/lc0/ dubbed experimental.

What exactly are these settings you're describing? I can't run a gpu based LC0 (yet).

Thanks,
CL

[Laskos]

In view of the many settings in LC0, I ran a deep CLOP on three big ones at the same time. CLOP is designed to tune for the best playing performance, not best tactics or anything. The old default settings were nearly catastrophic giving NN369 a mere 109/200 solved in my corrected WAC set. The new settings yield... 159/200.



You can try them without any work, by downloading the latest LC0 June 4 build at https://crem.xyz/lc0/ dubbed experimental.

Great! You mean your settings are the default settings of the latest LC0 experimental? And the test was at 10s/position?
I got similar results in tactics tuning on your suite WAC200, with my "extreme tactical settings". The problem was that it was degrading significantly positionally on my positional test suite Openings200.

NN373, GTX 1060 6GB, 10s/position

The default (31 May) LC0 tested in Poliglot on these suites was:
WAC200: 106/200
Openings200: 124/200

My "extreme tactical" LC0 (31 May) tested as:
WAC200: 156/200
Openings200: 106/200

Your settings with LC0 (4 June) experimental:
WAC200: 154/200
Openings200: 120/200

So, it seems that your settings from CLOP are not degrading positionally significantly, improving tactically greatly.
At ultra-fast games, my "extreme tactical" performed much worse than the old default. In games at 10'+ 10'' (comparable to time control used in test-suites solving), they both performed equally badly against Komodo 10.2, both 3.5/20. I will run now this experimental LC0 against Komodo 10.2 in 20 10' +10'' games (maybe first trying ultra-fast games). The sample will be small, but it's really a pain, these 20 games will take some 16 hours, partly because I use no adjudication, and some games can last very long time. LC0 can lose even completely drawn easy endgames, so I use no adjudication.
WIth the old default LC0, it had more chances to gain advantage in the opening against Komodo 10.2, which is a strong positional player too. But it often blundered elementarily, spoiling all its gains. With my "extreme tactical", it blundered less, but was often dominated positionally by Komodo 10.2. All in all, both came badly at 3.5/20, which gave me serious doubts about the scaling of LC0 (it performed on par or even worse than in fast games). When in Europe will be evening, I will report my results with your settings. Thanks for your efforts with CLOP!

[peter]

Hi Kai!

The default (31 May) LC0 tested in Poliglot on these suites was:
WAC200: 106/200
Openings200: 124/200

My "extreme tactical" LC0 (31 May) tested as:
WAC200: 156/200
Openings200: 106/200

Your settings with LC0 (4 June) experimental:
WAC200: 154/200
Openings200: 120/200

So, it seems that your settings from CLOP are not degrading positionally significantly, improving tactically greatly.
...
Thanks for your efforts with CLOP!

Sounds great, but hoping not to be called too negative or nosy in a thing, I still don't understand anything better than before, I yet ponder, if LCZero should not be called somewhat else but Zero since a little while already?
I mean, if human input is now to change Clop and other "Settings" daily, where is the big difference to AB- engines you add and change patches in the programming code manually to get better results?

Again, no offence meant, especially I enjoy seeing that even test position- testing has become modern again to adapt an engine to, not only to eng-eng-matches anymore.

Yet I have to remind to an old wording about test positions in German "Stellungstests testen die Teststellungen", which never ever was something negative to me, yet you always have to take care not to be too selective as for the test positions chosen, wanting to find the very best ones for the very best result for a special engine. That's the classical bias called self-fulfilling prophecy.

[Albert Silver]

In view of the many settings in LC0, I ran a deep CLOP on three big ones at the same time. CLOP is designed to tune for the best playing performance, not best tactics or anything. The old default settings were nearly catastrophic giving NN369 a mere 109/200 solved in my corrected WAC set. The new settings yield... 159/200.



You can try them without any work, by downloading the latest LC0 June 4 build at https://crem.xyz/lc0/ dubbed experimental.

Instead of running meaningless "tuning" try at least beating the default version with your "magnificent" tuned parameters.

Done.

Here is first match played at 30s+0.5s with default values:

Score of lc0-june2 vs Protector 1.9: 31 - 46 - 23 [0.425]
Elo difference: -52.51 +/- 60.80

100 of 100 games finished.

Here is score with the new settings:

Score of lc0-june4-utk vs Protector 1.9: 36 - 33 - 31 [0.515]
Elo difference: 10.43 +/- 57.01

100 of 100 games finished.

[Milos]

Hi Kai!

The default (31 May) LC0 tested in Poliglot on these suites was:
WAC200: 106/200
Openings200: 124/200

My "extreme tactical" LC0 (31 May) tested as:
WAC200: 156/200
Openings200: 106/200

Your settings with LC0 (4 June) experimental:
WAC200: 154/200
Openings200: 120/200

So, it seems that your settings from CLOP are not degrading positionally significantly, improving tactically greatly.
...
Thanks for your efforts with CLOP!

Sounds great, but hoping not to be called too negative or nosy in a thing, I still don't understand anything better than before, I yet ponder, if LCZero should not be called somewhat else but Zero since a little while already?
I mean, if human input is now to change Clop and other "Settings" daily, where is the big difference to AB- engines you add and change patches in the programming code manually to get better results?

Again, no offence meant, especially I enjoy seeing that even test position- testing has become modern again to adapt an engine to, not only to eng-eng-matches anymore.

Yet I have to remind to an old wording about test positions in German "Stellungstests testen die Teststellungen", which never ever was something negative to me, yet you always have to take care not to be too selective as for the test positions chosen, wanting to find the very best ones for the very best result for a special engine. That's the classical bias called self-fulfilling prophecy.

The only "zero" thing is AlphaZero (if you believe that crappy preprint anyway) was NN. Everything else was tuned extensively. Even NN was trained multiple times from scratch and only the best training result has been shown in that "paper".

[Milos]

In view of the many settings in LC0, I ran a deep CLOP on three big ones at the same time. CLOP is designed to tune for the best playing performance, not best tactics or anything. The old default settings were nearly catastrophic giving NN369 a mere 109/200 solved in my corrected WAC set. The new settings yield... 159/200.



You can try them without any work, by downloading the latest LC0 June 4 build at https://crem.xyz/lc0/ dubbed experimental.

Instead of running meaningless "tuning" try at least beating the default version with your "magnificent" tuned parameters.

Done.

Here is first match played at 30s+0.5s with default values:

Score of lc0-june2 vs Protector 1.9: 31 - 46 - 23 [0.425]
Elo difference: -52.51 +/- 60.80

100 of 100 games finished.

Here is score with the new settings:

Score of lc0-june4-utk vs Protector 1.9: 36 - 33 - 31 [0.515]
Elo difference: 10.43 +/- 57.01

100 of 100 games finished.

That looks ok but I am confused which options did you use, since difference between yours June 4 and "default" June 2 version is basically just FPUR increased from 0.2 to 0.9, which is totally opposite direction from before and very mild increase of PCUT from 3.1 to 3.4 or something.
Anyway difference is still within error margins so could be just luck or it just worked against particular engine. What I've thought you did is to test it against default LC0 settings for PCUT and FPUG which are for cudnn version something like 1.7 and 0.0.

[Albert Silver]

Instead of running meaningless "tuning" try at least beating the default version with your "magnificent" tuned parameters.

Done.

Here is first match played at 30s+0.5s with default values:

Score of lc0-june2 vs Protector 1.9: 31 - 46 - 23 [0.425]
Elo difference: -52.51 +/- 60.80

100 of 100 games finished.

Here is score with the new settings:

Score of lc0-june4-utk vs Protector 1.9: 36 - 33 - 31 [0.515]
Elo difference: 10.43 +/- 57.01

100 of 100 games finished.

That looks ok but I am confused which options did you use, since difference between yours June 4 and "default" June 2 version is basically just FPUR increased from 0.2 to 0.9, which is totally opposite direction from before and very mild increase of PCUT from 3.1 to 3.4 or something.
Anyway difference is still within error margins so could be just luck or it just worked against particular engine. What I've thought you did is to test it against default LC0 settings for PCUT and FPUG which are for cudnn version something like 1.7 and 0.0.

No, there is a third you did not notice which has a similarly strong effect: Policy Softmax Temperature. I myself did not know it until its author brought it to my attention in Discord.

[jp]

there is a third which has a similarly strong effect: Policy Softmax Temperature. I myself did not know it until its author brought it to my attention.

Did the author say in depth what this does, beyond what we can guess from the words?