Lc0 wins, but Stockfish is still the best?

[mwyoung]

I have a subscriber to my channel who loves Stockfish. And is very upset that Lc0 has overtaken Stockfish on my channel streams. I don't understand the emotional attachment to a chess program. As you would love and be a fan of a human chess player, but I do know it does exist.

My thoughts are no matter how good Lc0 performs. I feel CPU engines are superior at this point in time for many reasons.

First I don't believe the so called Leela Ratio is a fair testing standard. Because what does it cost to meet this standard.
This gives Lc0 a huge hardware price advantage. On my testing system. I run a 16 core CPU for $799. At the same time Lc0 plays on a $1500 RTX 2080 ti. Is Lc0 better, yes but Lc0 needs every bit of the 2080 ti. And at almost twice the price to best Stockfish.

I like when new programs and technology can advance chess. And I have no issue with Lc0 besting Stockfish or any program.

But from a practical point of view as a chess player. CPU engines are still the best for price to performance. Stockfish will play great on my phone. That I can take anywhere. Stockfish plays great on my 4 core cpu laptop that I take to tournaments.

At this point in time CPU based engines are still the best.

[Nay Lin Tun]

Those CPU/ GPU prices may change in next couple of years.

Regarding smartphone chess, are you aware that Huawei mate 10/20 pro support Neural Network processors. There is no android version of Leela yet but if chess is one of the reason in choosing smart phone, then Huawei is a big blow over his competitors samsung/apple/ google phones.

If Leela running on Hauwei beat SF in the same phone, then it would be a real reason to use NN in smart phones.

[Modern Times]

First I don't believe the so called Leela Ratio is a fair testing standard. Because what does it cost to meet this standard.
This gives Lc0 a huge hardware price advantage. On my testing system. I run a 16 core CPU for $799. At the same time Lc0 plays on a $1500 RTX 2080 ti. Is Lc0 better, yes but Lc0 needs every bit of the 2080 ti. And at almost twice the price to best Stockfish.

Very valid point. This article explains the Leela ration a bit:

http://archive.is/o2qMP

I don't believe in it but not just for cost reasons. It is based on Alpha Zero's nps vs Stockfish 8 nps. Experience of CPU engines has shown that all nps numbers are not created equal ! Some engines have big nps numbers and others much smaller, on the same CPU. And bigger is not always better. Programmers can count nodes differently. Trying to equalise CPU engines based on their nps would be bad enough, but trying to equalise GPU and CPU nps ? Highly questionable. I simply don't think there is a way to try and say a certain GPU s equal to a certain CPU. They are so vastly different.

[mwyoung]

First I don't believe the so called Leela Ratio is a fair testing standard. Because what does it cost to meet this standard.
This gives Lc0 a huge hardware price advantage. On my testing system. I run a 16 core CPU for $799. At the same time Lc0 plays on a $1500 RTX 2080 ti. Is Lc0 better, yes but Lc0 needs every bit of the 2080 ti. And at almost twice the price to best Stockfish.

Very valid point. This article explains the Leela ration a bit:

http://archive.is/o2qMP

I don't believe in it but not just for cost reasons. It is based on Alpha Zero's nps vs Stockfish 8 nps. Experience of CPU engines has shown that all nps numbers are not created equal ! Some engines have big nps numbers and others much smaller, on the same CPU. And bigger is not always better. Programmers can count nodes differently. Trying to equalise CPU engines based on their nps would be bad enough, but trying to equalise GPU and CPU nps ? Highly questionable. I simply don't think there is a way to try and say a certain GPU s equal to a certain CPU. They are so vastly different.

I agree, and I have made the same points many times about NPS. It has fallen on deaf hears. Hopefully price to performance will be easier to understand. Why the LR is not a fair testing standard.

You can not equalize GPU and CPU nps, but you can equalize price. This is what matters. Price to performance.

[Jouni]

And remember, that Stockfish dev is 1167872 bytes! How much is current Lc0? 100 megabytes + cuda things? SF should get 1/100 time handicap .

[Eduard]

I have a subscriber to my channel who loves Stockfish. And is very upset that Lc0 has overtaken Stockfish on my channel streams. I don't understand the emotional attachment to a chess program. As you would love and be a fan of a human chess player, but I do know it does exist.

My thoughts are no matter how good Lc0 performs. I feel CPU engines are superior at this point in time for many reasons.

First I don't believe the so called Leela Ratio is a fair testing standard. Because what does it cost to meet this standard.
This gives Lc0 a huge hardware price advantage. On my testing system. I run a 16 core CPU for $799. At the same time Lc0 plays on a $1500 RTX 2080 ti. Is Lc0 better, yes but Lc0 needs every bit of the 2080 ti. And at almost twice the price to best Stockfish.

I like when new programs and technology can advance chess. And I have no issue with Lc0 besting Stockfish or any program.

But from a practical point of view as a chess player. CPU engines are still the best for price to performance. Stockfish will play great on my phone. That I can take anywhere. Stockfish plays great on my 4 core cpu laptop that I take to tournaments.

At this point in time CPU based engines are still the best.

I agree with that.

[corres]

If one want to be correct that one have to give the circumstances of the test or tournament implying the time control and the used opening book too.
In the sense of chess power obviously there is no correct method to make comparison between a CPU determined hardware and a GPU or TPU determined hardware. From practical viewpoint now the Elo of Lc0 and Elo of Stockfish is near the same but both of them there is the own benefit and the own disadvantage against each other.
I think being fun is not a technical category and peoples who use engines for analysis do not choose engine based on emotions.

[corres]

If one want to be correct that one have to give the circumstances of the test or tournament implying the time control and the used opening book too.
In the sense of chess power obviously there is no correct method to make comparison between a CPU determined hardware and a GPU or TPU determined hardware. From practical viewpoint now the Elo of Lc0 and Elo of Stockfish is near the same but both of them there is the own benefit and the own disadvantage against each other.
I think being fun is not a technical category and peoples who use engines for analysis do not choose engine based on emotions.

Naturally "I think being FAN...."
Sorry.

[Laskos]

This gives Lc0 a huge hardware price advantage. On my testing system. I run a 16 core CPU for $799. At the same time Lc0 plays on a $1500 RTX 2080 ti. Is Lc0 better, yes but Lc0 needs every bit of the 2080 ti. And at almost twice the price to best Stockfish.

You can easily find an RTX 2080ti in $1200-$1250 range. Doesn't your CPU need a pretty large cooler to run consistently, without throttling on some very heavy and long CPU loads? That's another $70-$100 to almost $900 for CPU. GPUs don't need that. Also, RTX 2080ti doesn't seem to be the best deal around as GPU goes. I have my RTX 2070 for $550, 2.7 times cheaper than your expensive RTX 2080ti.

Let's compare apples to apples: Take the net 41687 (each net has its own speed curve, it is the latest I downloaded), put it on infinite from the standard opening position with "go". The engine should be Lc0 v0.21.1. Observe the speed after 10 million nodes. Mine is:

info depth 19 seldepth 57 time 232129 nodes 10456908 score cp 44 hashfull 246 nps 45047

I am curious what your 2.7 factor more expensive RTX 2080ti will show. If the scaling is good, two RTX 2070 would be a bit cheaper than one RTX 2080ti and I bet they would be faster.

As for CPU, I will buy this Summer or Autumn the third generation Ryzen 9 3850X, a 16 core CPU even stronger than your Threadripper 2950X, but for only $500.

All in all, Leela with latest T40 nets is very competitive with SF_dev strength-wise on fairly equal price-wise home-grade hardware.

[mwyoung]

This gives Lc0 a huge hardware price advantage. On my testing system. I run a 16 core CPU for $799. At the same time Lc0 plays on a $1500 RTX 2080 ti. Is Lc0 better, yes but Lc0 needs every bit of the 2080 ti. And at almost twice the price to best Stockfish.

You can easily find an RTX 2080ti in $1200-$1250 range. Doesn't your CPU need a pretty large cooler to run consistently, without throttling on some very heavy and long CPU loads? That's another $70-$100 to almost $900 for CPU. GPUs don't need that. Also, RTX 2080ti doesn't seem to be the best deal around as GPU goes. I have my RTX 2070 for $550, 2.7 times cheaper than your expensive RTX 2080ti.

Let's compare apples to apples: Take the net 41687 (each net has its own speed curve, it is the latest I downloaded), put it on infinite from the standard opening position with "go". The engine should be Lc0 v0.21.1. Observe the speed after 10 million nodes. Mine is:

info depth 19 seldepth 57 time 232129 nodes 10456908 score cp 44 hashfull 246 nps 45047

I am curious what your 2.7 factor more expensive RTX 2080ti will show. If the scaling is good, two RTX 2070 would be a bit cheaper than one RTX 2080ti and I bet they would be faster.

As for CPU, I will buy this Summer or Autumn the third generation Ryzen 9 3850X, a 16 core CPU even stronger than your Threadripper 2950X, but for only $500.

All in all, Leela with latest T40 nets is very competitive with SF_dev strength-wise on fairly equal price-wise home-grade hardware.

You can buy a RTX 2080 ti for $1200. But I would not recommend using it for Lc0. Or you will be another poster on here. Posting that the card is running Lc0 at 26kns. Or asking why is my gpu running at 70%.

I bought the card I did because it had the best cooler on the market. Lc0 is brutal on a graphics card running at 300 watts. And I run for hours or days at a time when testing.

The 3850x is rumor, but if released. I would not buy it for a testing system. Because it will only be dual channel memory. This is not good when you are feeding 16 cores and 32 threads of fun.

The new Threadrippers will also be releasing. And it will be 4 channel memory. This is what you need. If you are buying an AMD based chess computer.