New CLOP settings give Leela huge tactics boost

Discussion of anything and everything relating to chess playing software and machines.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
Albert Silver
Posts: 2860
Joined: Wed Mar 08, 2006 8:57 pm
Location: Rio de Janeiro, Brazil

New CLOP settings give Leela huge tactics boost

Post by Albert Silver » Mon Jun 04, 2018 7:02 pm

In view of the many settings in LC0, I ran a deep CLOP on three big ones at the same time. CLOP is designed to tune for the best playing performance, not best tactics or anything. The old default settings were nearly catastrophic giving NN369 a mere 109/200 solved in my corrected WAC set. The new settings yield... 159/200.

Image

You can try them without any work, by downloading the latest LC0 June 4 build at https://crem.xyz/lc0/ dubbed experimental.
"Tactics are the bricks and sticks that make up a game, but positional play is the architectural blueprint."

Milos
Posts: 3387
Joined: Wed Nov 25, 2009 12:47 am

Re: New CLOP settings give Leela huge tactics boost

Post by Milos » Mon Jun 04, 2018 8:13 pm

Albert Silver wrote:
Mon Jun 04, 2018 7:02 pm
In view of the many settings in LC0, I ran a deep CLOP on three big ones at the same time. CLOP is designed to tune for the best playing performance, not best tactics or anything. The old default settings were nearly catastrophic giving NN369 a mere 109/200 solved in my corrected WAC set. The new settings yield... 159/200.

Image

You can try them without any work, by downloading the latest LC0 June 4 build at https://crem.xyz/lc0/ dubbed experimental.
Instead of running meaningless "tuning" try at least beating the default version with your "magnificent" tuned parameters.

carldaman
Posts: 1717
Joined: Sat Jun 02, 2012 12:13 am

Re: New CLOP settings give Leela huge tactics boost

Post by carldaman » Mon Jun 04, 2018 11:52 pm

Albert Silver wrote:
Mon Jun 04, 2018 7:02 pm
In view of the many settings in LC0, I ran a deep CLOP on three big ones at the same time. CLOP is designed to tune for the best playing performance, not best tactics or anything. The old default settings were nearly catastrophic giving NN369 a mere 109/200 solved in my corrected WAC set. The new settings yield... 159/200.

Image

You can try them without any work, by downloading the latest LC0 June 4 build at https://crem.xyz/lc0/ dubbed experimental.
What exactly are these settings you're describing? I can't run a gpu based LC0 (yet).

Thanks,
CL

User avatar
Laskos
Posts: 9445
Joined: Wed Jul 26, 2006 8:21 pm
Full name: Kai Laskos

Re: New CLOP settings give Leela huge tactics boost

Post by Laskos » Tue Jun 05, 2018 3:01 am

Albert Silver wrote:
Mon Jun 04, 2018 7:02 pm
In view of the many settings in LC0, I ran a deep CLOP on three big ones at the same time. CLOP is designed to tune for the best playing performance, not best tactics or anything. The old default settings were nearly catastrophic giving NN369 a mere 109/200 solved in my corrected WAC set. The new settings yield... 159/200.

Image

You can try them without any work, by downloading the latest LC0 June 4 build at https://crem.xyz/lc0/ dubbed experimental.
Great! You mean your settings are the default settings of the latest LC0 experimental? And the test was at 10s/position?
I got similar results in tactics tuning on your suite WAC200, with my "extreme tactical settings". The problem was that it was degrading significantly positionally on my positional test suite Openings200.

NN373, GTX 1060 6GB, 10s/position

The default (31 May) LC0 tested in Poliglot on these suites was:
WAC200: 106/200
Openings200: 124/200

My "extreme tactical" LC0 (31 May) tested as:
WAC200: 156/200
Openings200: 106/200

Your settings with LC0 (4 June) experimental:
WAC200: 154/200
Openings200: 120/200

So, it seems that your settings from CLOP are not degrading positionally significantly, improving tactically greatly.
At ultra-fast games, my "extreme tactical" performed much worse than the old default. In games at 10'+ 10'' (comparable to time control used in test-suites solving), they both performed equally badly against Komodo 10.2, both 3.5/20. I will run now this experimental LC0 against Komodo 10.2 in 20 10' +10'' games (maybe first trying ultra-fast games). The sample will be small, but it's really a pain, these 20 games will take some 16 hours, partly because I use no adjudication, and some games can last very long time. LC0 can lose even completely drawn easy endgames, so I use no adjudication.
WIth the old default LC0, it had more chances to gain advantage in the opening against Komodo 10.2, which is a strong positional player too. But it often blundered elementarily, spoiling all its gains. With my "extreme tactical", it blundered less, but was often dominated positionally by Komodo 10.2. All in all, both came badly at 3.5/20, which gave me serious doubts about the scaling of LC0 (it performed on par or even worse than in fast games). When in Europe will be evening, I will report my results with your settings. Thanks for your efforts with CLOP!

peter
Posts: 1782
Joined: Sat Feb 16, 2008 6:38 am
Full name: Peter Martan

Re: New CLOP settings give Leela huge tactics boost

Post by peter » Tue Jun 05, 2018 5:19 am

Hi Kai!
Laskos wrote:
Tue Jun 05, 2018 3:01 am
The default (31 May) LC0 tested in Poliglot on these suites was:
WAC200: 106/200
Openings200: 124/200

My "extreme tactical" LC0 (31 May) tested as:
WAC200: 156/200
Openings200: 106/200

Your settings with LC0 (4 June) experimental:
WAC200: 154/200
Openings200: 120/200

So, it seems that your settings from CLOP are not degrading positionally significantly, improving tactically greatly.
...
Thanks for your efforts with CLOP!
Sounds great, but hoping not to be called too negative or nosy in a thing, I still don't understand anything better than before, I yet ponder, if LCZero should not be called somewhat else but Zero since a little while already?
I mean, if human input is now to change Clop and other "Settings" daily, where is the big difference to AB- engines you add and change patches in the programming code manually to get better results?

Again, no offence meant, especially I enjoy seeing that even test position- testing has become modern again to adapt an engine to, not only to eng-eng-matches anymore.

Yet I have to remind to an old wording about test positions in German "Stellungstests testen die Teststellungen", which never ever was something negative to me, yet you always have to take care not to be too selective as for the test positions chosen, wanting to find the very best ones for the very best result for a special engine. That's the classical bias called self-fulfilling prophecy.
:)
Last edited by peter on Tue Jun 05, 2018 5:36 am, edited 1 time in total.
Peter.

Albert Silver
Posts: 2860
Joined: Wed Mar 08, 2006 8:57 pm
Location: Rio de Janeiro, Brazil

Re: New CLOP settings give Leela huge tactics boost

Post by Albert Silver » Tue Jun 05, 2018 5:29 am

Milos wrote:
Mon Jun 04, 2018 8:13 pm
Albert Silver wrote:
Mon Jun 04, 2018 7:02 pm
In view of the many settings in LC0, I ran a deep CLOP on three big ones at the same time. CLOP is designed to tune for the best playing performance, not best tactics or anything. The old default settings were nearly catastrophic giving NN369 a mere 109/200 solved in my corrected WAC set. The new settings yield... 159/200.

Image

You can try them without any work, by downloading the latest LC0 June 4 build at https://crem.xyz/lc0/ dubbed experimental.
Instead of running meaningless "tuning" try at least beating the default version with your "magnificent" tuned parameters.
Done.

Here is first match played at 30s+0.5s with default values:

Score of lc0-june2 vs Protector 1.9: 31 - 46 - 23 [0.425]
Elo difference: -52.51 +/- 60.80

100 of 100 games finished.

Here is score with the new settings:

Score of lc0-june4-utk vs Protector 1.9: 36 - 33 - 31 [0.515]
Elo difference: 10.43 +/- 57.01

100 of 100 games finished.
"Tactics are the bricks and sticks that make up a game, but positional play is the architectural blueprint."

Milos
Posts: 3387
Joined: Wed Nov 25, 2009 12:47 am

Re: New CLOP settings give Leela huge tactics boost

Post by Milos » Tue Jun 05, 2018 8:24 am

peter wrote:
Tue Jun 05, 2018 5:19 am
Hi Kai!
Laskos wrote:
Tue Jun 05, 2018 3:01 am
The default (31 May) LC0 tested in Poliglot on these suites was:
WAC200: 106/200
Openings200: 124/200

My "extreme tactical" LC0 (31 May) tested as:
WAC200: 156/200
Openings200: 106/200

Your settings with LC0 (4 June) experimental:
WAC200: 154/200
Openings200: 120/200

So, it seems that your settings from CLOP are not degrading positionally significantly, improving tactically greatly.
...
Thanks for your efforts with CLOP!
Sounds great, but hoping not to be called too negative or nosy in a thing, I still don't understand anything better than before, I yet ponder, if LCZero should not be called somewhat else but Zero since a little while already?
I mean, if human input is now to change Clop and other "Settings" daily, where is the big difference to AB- engines you add and change patches in the programming code manually to get better results?

Again, no offence meant, especially I enjoy seeing that even test position- testing has become modern again to adapt an engine to, not only to eng-eng-matches anymore.

Yet I have to remind to an old wording about test positions in German "Stellungstests testen die Teststellungen", which never ever was something negative to me, yet you always have to take care not to be too selective as for the test positions chosen, wanting to find the very best ones for the very best result for a special engine. That's the classical bias called self-fulfilling prophecy.
:)
The only "zero" thing is AlphaZero (if you believe that crappy preprint anyway) was NN. Everything else was tuned extensively. Even NN was trained multiple times from scratch and only the best training result has been shown in that "paper".

Milos
Posts: 3387
Joined: Wed Nov 25, 2009 12:47 am

Re: New CLOP settings give Leela huge tactics boost

Post by Milos » Tue Jun 05, 2018 12:01 pm

Albert Silver wrote:
Tue Jun 05, 2018 5:29 am
Milos wrote:
Mon Jun 04, 2018 8:13 pm
Albert Silver wrote:
Mon Jun 04, 2018 7:02 pm
In view of the many settings in LC0, I ran a deep CLOP on three big ones at the same time. CLOP is designed to tune for the best playing performance, not best tactics or anything. The old default settings were nearly catastrophic giving NN369 a mere 109/200 solved in my corrected WAC set. The new settings yield... 159/200.

Image

You can try them without any work, by downloading the latest LC0 June 4 build at https://crem.xyz/lc0/ dubbed experimental.
Instead of running meaningless "tuning" try at least beating the default version with your "magnificent" tuned parameters.
Done.

Here is first match played at 30s+0.5s with default values:

Score of lc0-june2 vs Protector 1.9: 31 - 46 - 23 [0.425]
Elo difference: -52.51 +/- 60.80

100 of 100 games finished.

Here is score with the new settings:

Score of lc0-june4-utk vs Protector 1.9: 36 - 33 - 31 [0.515]
Elo difference: 10.43 +/- 57.01

100 of 100 games finished.
That looks ok but I am confused which options did you use, since difference between yours June 4 and "default" June 2 version is basically just FPUR increased from 0.2 to 0.9, which is totally opposite direction from before and very mild increase of PCUT from 3.1 to 3.4 or something.
Anyway difference is still within error margins so could be just luck or it just worked against particular engine. What I've thought you did is to test it against default LC0 settings for PCUT and FPUG which are for cudnn version something like 1.7 and 0.0.

Albert Silver
Posts: 2860
Joined: Wed Mar 08, 2006 8:57 pm
Location: Rio de Janeiro, Brazil

Re: New CLOP settings give Leela huge tactics boost

Post by Albert Silver » Tue Jun 05, 2018 12:30 pm

Milos wrote:
Tue Jun 05, 2018 12:01 pm
Albert Silver wrote:
Tue Jun 05, 2018 5:29 am
Milos wrote:
Mon Jun 04, 2018 8:13 pm

Instead of running meaningless "tuning" try at least beating the default version with your "magnificent" tuned parameters.
Done.

Here is first match played at 30s+0.5s with default values:

Score of lc0-june2 vs Protector 1.9: 31 - 46 - 23 [0.425]
Elo difference: -52.51 +/- 60.80

100 of 100 games finished.

Here is score with the new settings:

Score of lc0-june4-utk vs Protector 1.9: 36 - 33 - 31 [0.515]
Elo difference: 10.43 +/- 57.01

100 of 100 games finished.
That looks ok but I am confused which options did you use, since difference between yours June 4 and "default" June 2 version is basically just FPUR increased from 0.2 to 0.9, which is totally opposite direction from before and very mild increase of PCUT from 3.1 to 3.4 or something.
Anyway difference is still within error margins so could be just luck or it just worked against particular engine. What I've thought you did is to test it against default LC0 settings for PCUT and FPUG which are for cudnn version something like 1.7 and 0.0.
No, there is a third you did not notice which has a similarly strong effect: Policy Softmax Temperature. I myself did not know it until its author brought it to my attention in Discord.
"Tactics are the bricks and sticks that make up a game, but positional play is the architectural blueprint."

jp
Posts: 818
Joined: Mon Apr 23, 2018 5:54 am

Re: New CLOP settings give Leela huge tactics boost

Post by jp » Tue Jun 05, 2018 1:15 pm

Albert Silver wrote:
Tue Jun 05, 2018 12:30 pm
there is a third which has a similarly strong effect: Policy Softmax Temperature. I myself did not know it until its author brought it to my attention.
Did the author say in depth what this does, beyond what we can guess from the words?

Post Reply