Lc0 will hardly ever win TCEC or CCCC?

Discussion of anything and everything relating to chess playing software and machines.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
User avatar
Laskos
Posts: 10612
Joined: Wed Jul 26, 2006 8:21 pm
Full name: Kai Laskos

Lc0 will hardly ever win TCEC or CCCC?

Post by Laskos » Wed Jul 22, 2020 8:48 am

Best SF NNUE nets (Sergio's of today, for example) with GK binary seem to hold against Lc0 even in my very GPU-skewed conditions.

4 i7 cores
RTX 2070 GPU
Old "Leela Ratio" of about 3.0 --- very GPU favorable.

60s + 0.6s

Code: Select all

Score of Sergio_22.07_0944 NNUE vs Lc0 SV_1541: 27 - 28 - 45  [0.495] 100

...      Sergio_22.07_0944 NNUE playing White: 27 - 2 - 21  [0.750] 50
...      Sergio_22.07_0944 NNUE playing Black: 0 - 26 - 24  [0.240] 50
...      White vs Black: 53 - 2 - 45  [0.755] 100
Elo difference: -3.5 +/- 50.8, LOS: 44.6 %, DrawRatio: 45.0 %
Finished match

SF_dev is beaten heavily in these conditions by Lc0, but SF NNUE holds its ground against Lc0,
If SF NNUE holds in these conditions (3.0 Leela Ratio), then in TCEC and CCCC, where the old "Leela Ratio" is most often about ~1, Lc0 will be beaten heavily by SF NNUE. In TCEC 18, even SF_dev managed to beat it.

User avatar
cdani
Posts: 2172
Joined: Sat Jan 18, 2014 9:24 am
Location: Andorra
Contact:

Re: Lc0 will hardly ever win TCEC or CCCC?

Post by cdani » Wed Jul 22, 2020 10:39 am

A fast test between two of the latest sergio's nets, 15+0.1

Code: Select all

   # PLAYER                       : RATING  ERROR   POINTS  PLAYED    (%)
   1   20200722-1517    : 2866.5    7.2    340.5     643   53.0%
   2   20200722-0359    : 2845.5    7.2    302.5     643   47.0%
Same best sergio's net against GK best (that was already better than Stockfish dev), 20 + 0.1:

Code: Select all

   # PLAYER                       : RATING  ERROR   POINTS  PLAYED    (%)
   1 20200722-1517     : 2875.4    7.1    357.0     644   55.4%
   2 gekehenker2706net : 2836.6    7.1    287.0     644   44.6%
Congratulations Sergio!! Your net is terrible strong!

JJJ
Posts: 1316
Joined: Sat Apr 19, 2014 11:47 am

Re: Lc0 will hardly ever win TCEC or CCCC?

Post by JJJ » Wed Jul 22, 2020 12:52 pm

Laskos wrote:
Wed Jul 22, 2020 8:48 am
Best SF NNUE nets (Sergio's of today, for example) with GK binary seem to hold against Lc0 even in my very GPU-skewed conditions.

4 i7 cores
RTX 2070 GPU
Old "Leela Ratio" of about 3.0 --- very GPU favorable.

60s + 0.6s

Code: Select all

Score of Sergio_22.07_0944 NNUE vs Lc0 SV_1541: 27 - 28 - 45  [0.495] 100

...      Sergio_22.07_0944 NNUE playing White: 27 - 2 - 21  [0.750] 50
...      Sergio_22.07_0944 NNUE playing Black: 0 - 26 - 24  [0.240] 50
...      White vs Black: 53 - 2 - 45  [0.755] 100
Elo difference: -3.5 +/- 50.8, LOS: 44.6 %, DrawRatio: 45.0 %
Finished match

SF_dev is beaten heavily in these conditions by Lc0, but SF NNUE holds its ground against Lc0,
If SF NNUE holds in these conditions (3.0 Leela Ratio), then in TCEC and CCCC, where the old "Leela Ratio" is most often about ~1, Lc0 will be beaten heavily by SF NNUE. In TCEC 18, even SF_dev managed to beat it.
In CCC, lczero won nicely against Stockfish NNUE with a nice margin. So who knows ?

Raphexon
Posts: 299
Joined: Sun Mar 17, 2019 11:00 am
Full name: Henk Drost

Re: Lc0 will hardly ever win TCEC or CCCC?

Post by Raphexon » Wed Jul 22, 2020 12:58 pm

JJJ wrote:
Wed Jul 22, 2020 12:52 pm
Laskos wrote:
Wed Jul 22, 2020 8:48 am
Best SF NNUE nets (Sergio's of today, for example) with GK binary seem to hold against Lc0 even in my very GPU-skewed conditions.

4 i7 cores
RTX 2070 GPU
Old "Leela Ratio" of about 3.0 --- very GPU favorable.

60s + 0.6s

Code: Select all

Score of Sergio_22.07_0944 NNUE vs Lc0 SV_1541: 27 - 28 - 45  [0.495] 100

...      Sergio_22.07_0944 NNUE playing White: 27 - 2 - 21  [0.750] 50
...      Sergio_22.07_0944 NNUE playing Black: 0 - 26 - 24  [0.240] 50
...      White vs Black: 53 - 2 - 45  [0.755] 100
Elo difference: -3.5 +/- 50.8, LOS: 44.6 %, DrawRatio: 45.0 %
Finished match

SF_dev is beaten heavily in these conditions by Lc0, but SF NNUE holds its ground against Lc0,
If SF NNUE holds in these conditions (3.0 Leela Ratio), then in TCEC and CCCC, where the old "Leela Ratio" is most often about ~1, Lc0 will be beaten heavily by SF NNUE. In TCEC 18, even SF_dev managed to beat it.
In CCC, lczero won nicely against Stockfish NNUE with a nice margin. So who knows ?
Untested net from Jjoshua that's behind a paywall on Patreon.
Maybe the extra slowdown is also an issue inherent to 384x sized net on that hardware.
Maybe VRM is overheating from AVX usage.


Vondele's 250 thread test (although SSS) indicated no problems:
https://github.com/official-stockfish/S ... -660519116

the_real_greco
Posts: 45
Joined: Thu Sep 19, 2019 3:55 am
Full name: Andy!

Re: Lc0 will hardly ever win TCEC or CCCC?

Post by the_real_greco » Wed Jul 22, 2020 9:45 pm

Though I wasn't part of the technical discussions, I can confirm that the Lc0-SFNNUE match had some compile issues- we never got instructions to set a flag, or something. And who knows how fast a jjosh net should be?

So, this is your alert not to use that result.

Right now we're running a three-way, 3|2 match among Stockfish, Stockfish+NNUE, and Lc0 where everyone is running quite nicely. And this second one can be used as strength evidence, if there's still any doubt about SFNNUE.

Milos
Posts: 3832
Joined: Wed Nov 25, 2009 12:47 am

Re: Lc0 will hardly ever win TCEC or CCCC?

Post by Milos » Wed Jul 22, 2020 10:11 pm

CCC matches are anyway totally not representative of anything. Lc0 has a ridiculous advantage in hardware (CPU hardware is a total joke, weaker than Ryzen 3970X), plus TC is the best possible one for Lc0 (in shorter and longer TC SF scales better).

Raphexon
Posts: 299
Joined: Sun Mar 17, 2019 11:00 am
Full name: Henk Drost

Re: Lc0 will hardly ever win TCEC or CCCC?

Post by Raphexon » Wed Jul 22, 2020 10:16 pm

Milos wrote:
Wed Jul 22, 2020 10:11 pm
CCC matches are anyway totally not representative of anything. Lc0 has a ridiculous advantage in hardware (CPU hardware is a total joke, weaker than Ryzen 3970X), plus TC is the best possible one for Lc0 (in shorter and longer TC SF scales better).
They're still double Xeon plats...

Also TC should suit Stockfish, every other TC is a joke and a scam. :lol:

Milos
Posts: 3832
Joined: Wed Nov 25, 2009 12:47 am

Re: Lc0 will hardly ever win TCEC or CCCC?

Post by Milos » Wed Jul 22, 2020 11:04 pm

Raphexon wrote:
Wed Jul 22, 2020 10:16 pm
They're still double Xeon plats...
Yes, they are glorified overpriced pieces of junk. Probably the worst performance/price ratio CPUs in recent history.

Dann Corbit
Posts: 11510
Joined: Wed Mar 08, 2006 7:57 pm
Location: Redmond, WA USA
Contact:

Re: Lc0 will hardly ever win TCEC or CCCC?

Post by Dann Corbit » Thu Jul 23, 2020 5:44 am

Everything is a moving target (all the mentioned engines) and so it is very hard to make predictions.

I guess the ultimate winner will be ...
Something new. At some point anyway.
It quite often turns out that way.

We get surprised and we get surprised and somehow, even after being surprised over and over and over, we get surprised again. It's pretty strange when you think about it.

Watch Olithink become world champion. Of course by then it might be bloated to nearly 100k.
:twisted:
Taking ideas is not a vice, it is a virtue. We have another word for this. It is called learning.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.

Milos
Posts: 3832
Joined: Wed Nov 25, 2009 12:47 am

Re: Lc0 will hardly ever win TCEC or CCCC?

Post by Milos » Thu Jul 23, 2020 1:10 pm

Dann Corbit wrote:
Thu Jul 23, 2020 5:44 am
Everything is a moving target (all the mentioned engines) and so it is very hard to make predictions.

I guess the ultimate winner will be ...
Something new. At some point anyway.
It quite often turns out that way.

We get surprised and we get surprised and somehow, even after being surprised over and over and over, we get surprised again. It's pretty strange when you think about it.

Watch Olithink become world champion. Of course by then it might be bloated to nearly 100k.
:twisted:
Well it'll be over for quite some period. MCTS and big nets are not the way to go as it has been known for a while. And A/B with handcrafted eval is not using hardware in the best possible way. It's been clear for a while that A/B with fast small NN eval is a way to go. We lacked good implementation of it (we can now see how bad and "noob" was Giraffe NN for example). Now it's there. Once main SF devs embrace it and start tuning search together with eval, sky is the limit. This is a real paradigm shift, not A0 or Lc0.
Plus there is another aspect. New CPUs will be having more and more support for vector INT8 and BFLOAT16 multiplication operations. So running things on GPUs is not gonna make advantage.

Post Reply