Leela Chess Zero 42565 vs Stockfish 140619

Discussion of computer chess matches and engine tournaments.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
mwyoung
Posts: 1642
Joined: Wed May 12, 2010 8:00 pm

Re: Leela Chess Zero 42565 vs Stockfish 140619

Post by mwyoung » Thu Jun 20, 2019 5:50 pm

Nordlandia wrote:
Thu Jun 20, 2019 4:00 pm
MikeB: like i said - an exercise in futility :)

"It is still an exercise in futility. Let's say the move prediction is 50%. Both of your engines would still be reaching the correct response line half of the time in the time control and in the other half will have wasted considerable time finding the wrong move for nothing to gain from it at only half of the potential depth. Your engines might agree on the same predicted reply at 15ply for example but may change course once 30 ply is reached. An exercise in futility."
Yep. You have to ask the cost and return. It cost stockfish 6% and lc0 nothing. But stockfish can take advantage of the CPU overclock, but not lc0. As the GPU is the bottle neck. Not the CPU.
Professing themselves to be wise, they became fools,
Take on me. foes 0

mwyoung
Posts: 1642
Joined: Wed May 12, 2010 8:00 pm

Re: Leela Chess Zero 42565 vs Stockfish 140619

Post by mwyoung » Fri Jun 21, 2019 7:42 pm

ankan wrote:
Thu Jun 20, 2019 3:19 pm
mwyoung wrote:
Tue Jun 18, 2019 1:23 am
Laskos wrote:
Mon Jun 17, 2019 3:26 am
hgm wrote:
Sun Jun 16, 2019 9:51 pm
Pondering makes sense when playing against Leela on a many-core machine, as Leela wouldn't use many CPU threads while thinking, and all other threads would then go to waste. Likewise the GPUs would be idle during Stockfish' turn when Leela is not pondering.

If Leela need two unshared cores, you can just set an affinity for it for hyper-threads 0-3. You can then set affinity for Stockfish to the remaining HT. That way they won't compete for cores. They might still compete for memory bandwidth, though; not sure how important that is for Leela.

I don't see why Leela couldn't run its two threads on the same physical core. That could of course be a disadvantage compared to running 2 threads on 2 physical cores, but that also holds for Stockfish' threads sharing physical cores. Yet some people claim that hyper-threading is beneficial compared to running 1 thread per physical core, and testing both under conditions with 2 active HT per core does not seem particularly unfair. It is just like all HT are somewhat slower physical cores. This doesn't involve any scheduling, so it should not be noisy.

Of course if you don't also reserve some cores for the OS, that would cause noise.
This creature is running 34 threads on a 16 core machine, and it IS affecting heavily the engines, much more Leela than SF. I don't know, I have to perform plain experiment to show black on paper what is obvious:

4 cores i7 CPU, 8 logical cores.
Leela on RTX 2070 GPU using 2 threads.

1/ Leela to 1 million nodes on 2 threads (using one of the latest nets) without the interference from SF:

info depth 16 seldepth 52 time 35832 nodes 1021332 score cp 27 hashfull 481 nps 28503
info depth 16 seldepth 51 time 35866 nodes 1017115 score cp 27 hashfull 481 nps 28358
info depth 16 seldepth 52 time 36044 nodes 1027829 score cp 27 hashfull 483 nps 28515

Very high stability, within 0.5% deviation speeds.


2/ SF on 8 threads AND Leela to 1 million nodes on 2 threads:

info depth 16 seldepth 53 time 43960 nodes 1058510 score cp 27 hashfull 498 nps 24078
info depth 16 seldepth 52 time 48555 nodes 1017708 score cp 27 hashfull 480 nps 20959
info depth 16 seldepth 52 time 43277 nodes 999935 score cp 27 hashfull 474 nps 23105
info depth 16 seldepth 52 time 58460 nodes 1063476 score cp 27 hashfull 498 nps 18191

The speeds on average are some 30% lower and they are very erratic, some 20-30% deviation on average from one run to another.
The issue would be even graver with his 2080ti, as it uses a bit more of CPU resources than mine 2070. And more severe in shorter runs, as there are bursts of slowdowns. All in all, this creature's tests and posts here are plain garbage.
You know are comparing a 4 core i7. To a 2950x designed to run heavy process multi threaded workloads. With huge L1, L2, and L3 cache, and quad channel 8 dimm memory. With a much better memory controller, and process and thread scheduler. In other words a much more advanced architecture to run this kind of workload.

I am just saying :lol:
Can you try similar experiment on your system (2950x)?
Shouldn't take much time.

Run lc0 with 2 threads without SF running and check nps after 1m nodes. Then keep SF running with 32 threads in the background (e.g: analysis mode/go infinite) AND run lc0 with 2 threads and check lc0 nps after 1m nodes?
I am curious to know if your high end system is really so much better.
I did not forget your test. I just got back from vacation. Here is the results of your test you ask me to run. I ran them 5 times each, and lc0 is very consistent.

Lc0 ran by itself 26s
Lc0 ran with Stockfish 30 threads 26s
Lc0 ran with Stockfish 32 threads 26s.

Now I guess the reason I have been getting all this flak is some want to know my settings. And I would have told them directly if I was asked. I find it hard to help a community that attacks first, and then wants me to help them.

And It is a combination of settings and hardware. So yes hardware is a big reason. But you must know the simple settings, and have big enough hardware to support the setting. And I have told them exactly what I am doing, but they don't understand, or don't want to understand.

Hint: Again Lc0 is not a Ab engine. And threads are not processes.

What are my settings?
Professing themselves to be wise, they became fools,
Take on me. foes 0

mwyoung
Posts: 1642
Joined: Wed May 12, 2010 8:00 pm

Re: Leela Chess Zero 42565 vs Stockfish 140619

Post by mwyoung » Fri Jun 21, 2019 11:49 pm

Fact check:

Laskos Quotes:

Omg, I think people told you dozens of times --- why ponder = on? 32 threaded SF on 16 core machine ponder on? Ponder = on is completely wrecking your games. You would better use 1kW electric power on a fridge for ice creams.

Someone here is a vivid example that this forum had a good quantity of absolute imbeciles since its beginnings.

Leave this useless test and thread, here is one proceeding right now using similar hardware, but in correct conditions (ponder = on):

this creature is running 34 threads on a 16 core machine, and it IS affecting heavily the engines, much more Leela than SF. I don't know, I have to perform plain experiment to show black on paper what is obvious:

4 cores i7 CPU, 8 logical cores.
Leela on RTX 2070 GPU using 2 threads.

1/ Leela to 1 million nodes on 2 threads (using one of the latest nets) without the interference from SF:

info depth 16 seldepth 52 time 35832 nodes 1021332 score cp 27 hashfull 481 nps 28503
info depth 16 seldepth 51 time 35866 nodes 1017115 score cp 27 hashfull 481 nps 28358
info depth 16 seldepth 52 time 36044 nodes 1027829 score cp 27 hashfull 483 nps 28515

Very high stability, within 0.5% deviation speeds.


2/ SF on 8 threads AND Leela to 1 million nodes on 2 threads:

info depth 16 seldepth 53 time 43960 nodes 1058510 score cp 27 hashfull 498 nps 24078
info depth 16 seldepth 52 time 48555 nodes 1017708 score cp 27 hashfull 480 nps 20959
info depth 16 seldepth 52 time 43277 nodes 999935 score cp 27 hashfull 474 nps 23105
info depth 16 seldepth 52 time 58460 nodes 1063476 score cp 27 hashfull 498 nps 18191

The speeds on average are some 30% lower and they are very erratic, some 20-30% deviation on average from one run to another.
The issue would be even graver with his 2080ti, as it uses a bit more of CPU resources than mine 2070. And more severe in shorter runs, as there are bursts of slowdowns. All in all, this creature's tests and posts here are plain garbage.


Laskos -- "Ponder = on is completely wrecking your games."


Facts:
Lc0 is losing nothing playing ponder on matches. Stockfish is losing 6% CPU utilization running with Lc0 in ponder on matches.
But Stockfish has full access to the overclocked 16 core CPU running at 4.225 Ghz overclocked from the stock speed of 3.8 Ghz. With a ponder guess rate of over 57%. This is a win win for both Lc0 and Stockfish.

When playing ponder off the guess rate is of course 0%. As there is no thinking as the other engine calculates the move. And Stockfish has only 6% more CPU run time.

So in other words for the cost of 6% to Stockfish. Both programs can think as the other engines calculates with a ponder guess rate of over 57%.

--ankan wrote:
Can you try similar experiment on your system (2950x)?
Shouldn't take much time.

Run lc0 with 2 threads without SF running and check nps after 1m nodes. Then keep SF running with 32 threads in the background (e.g: analysis mode/go infinite) AND run lc0 with 2 threads and check lc0 nps after 1m nodes?
I am curious to know if your high end system is really so much better.

Results of the ankan test:
Lc0 ran by itself 26s
Lc0 ran with Stockfish 30 threads 26s
Lc0 ran with Stockfish 32 threads 26s.
--

Random sample games for Ponder on match games played between Lc0 2 theads vs Stockfish 32 threads. Game 25, 50, 100, 125 of latest match.

Lc0 Stats:

Game 25 W=18.9 plies; 368kN/s; 415 TBAs
Game 50 B=15.2 plies; 63kN/s; 32,903 TBAs
Game 100 B=23.0 plies; 74kN/s; 63,375 TBAs
Game 125 W=17.3 plies; 175kN/s; 282 TBAs

Stockfish Stats:

Game 25 B=53.3 plies; 37,962kN/s; 427,631,262 TBAs
Game 50 W=37.6 plies; 34,512kN/s; 521,796,329 TBAs
Game 100 W=45.5 plies; 40,942kN/s; 358,880,803 TBAs
Game 125 B=41.0 plies; 34,527kN/s; 107,565,908 TBAs





Professing themselves to be wise, they became fools,
Take on me. foes 0

Hugo
Posts: 775
Joined: Tue Dec 01, 2009 10:10 am

Re: Leela Chess Zero 42565 vs Stockfish 140619

Post by Hugo » Sat Jun 22, 2019 10:01 am

this thread is already awkward
nothing will change

mwyoung
Posts: 1642
Joined: Wed May 12, 2010 8:00 pm

Re: Leela Chess Zero 42565 vs Stockfish 140619

Post by mwyoung » Sat Jun 22, 2019 10:38 am

Hugo wrote:
Sat Jun 22, 2019 10:01 am
this thread is already awkward
nothing will change
:roll:
Professing themselves to be wise, they became fools,
Take on me. foes 0

User avatar
Nordlandia
Posts: 2477
Joined: Fri Sep 25, 2015 7:38 pm
Location: Sortland, Norway

Re: Leela Chess Zero 42565 vs Stockfish 140619

Post by Nordlandia » Sat Jun 22, 2019 11:25 am

mwyoung: swallow your pride and make the necessary adjustements, then everyone is pleased.

mwyoung
Posts: 1642
Joined: Wed May 12, 2010 8:00 pm

Re: Leela Chess Zero 42565 vs Stockfish 140619

Post by mwyoung » Sat Jun 22, 2019 12:00 pm

Nordlandia wrote:
Sat Jun 22, 2019 11:25 am
mwyoung: swallow your pride and make the necessary adjustements, then everyone is pleased.
What adjustments?

And who will be pleased?

Who are you speaking for?

And who thinks you can dictate other member's settings?
Professing themselves to be wise, they became fools,
Take on me. foes 0

User avatar
Nordlandia
Posts: 2477
Joined: Fri Sep 25, 2015 7:38 pm
Location: Sortland, Norway

Re: Leela Chess Zero 42565 vs Stockfish 140619

Post by Nordlandia » Sat Jun 22, 2019 12:30 pm

What i'm asking for is that you adjust SF threads to 29.

mwyoung
Posts: 1642
Joined: Wed May 12, 2010 8:00 pm

Re: Leela Chess Zero 42565 vs Stockfish 140619

Post by mwyoung » Sat Jun 22, 2019 12:55 pm

Nordlandia wrote:
Sat Jun 22, 2019 12:30 pm
What i'm asking for is that you adjust SF threads to 29.
Why do you want me to slow down stockfish? Because that is what your settings will do to stockfish on my 2950X. I am not running a 4 core I7.

I don't need to carve out 3 threads for Lc0. When Lc0 only needs 6% of the CPU. That would be like giving grandma a dragster to pickup food at the grocery store. All the heavy lifting is done by the GPU.

So the settings will stay, and the answer is no.

And you should know this about Lc0.

Do you even test lc0?

What equipment are you running?
Professing themselves to be wise, they became fools,
Take on me. foes 0

User avatar
Nordlandia
Posts: 2477
Joined: Fri Sep 25, 2015 7:38 pm
Location: Sortland, Norway

Re: Leela Chess Zero 42565 vs Stockfish 140619

Post by Nordlandia » Sat Jun 22, 2019 1:15 pm

mwyoung wrote:
Sat Jun 22, 2019 12:55 pm
Nordlandia wrote:
Sat Jun 22, 2019 12:30 pm
What i'm asking for is that you adjust SF threads to 29.
Why do you want me to slow down stockfish? Because that is what your settings will do to stockfish on my 2950X. I am not running a 4 core I7.

I don't need to carve out 3 threads for Lc0. When Lc0 only needs 6% of the CPU. That would be like giving grandma a dragster to pickup food at the grocery store. All the heavy lifting is done by the GPU.

So the settings will stay, and the answer is no.

And you should know this about Lc0.

Do you even test lc0?

What equipment are you running?
i7-5960X 4.5GHz 8-core (HT disabled) GTX 1070 Ti.
Now in case on ponder matches on my system i allocate 6 cores for SF and 1-core for Lc0 and 1 for "OS buffer"

One guy said that lc0 only need 1-core for GTX 1070 Ti. Single core is strong enough to feed it.

Post Reply