How to find your Leela Ratio - for Windows users w/ GPU(s)

Discussion of anything and everything relating to chess playing software and machines.

Moderators: hgm, Harvey Williamson, bob

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
jp
Posts: 706
Joined: Mon Apr 23, 2018 5:54 am

Re: How to find your Leela Ratio - for Windows users w/ GPU(s)

Post by jp » Thu Feb 28, 2019 1:53 pm

Mike S. wrote:
Thu Feb 28, 2019 1:44 pm
I think the whole concept of "Leela Ratio" is useless. You'll take your best buy anyway.
Useful only for comparing Lc with A0 (but then you'd also need to reproduce their other conditions and settings, which you probably can only guess at).
Bad name, too. It's got nothing to do with Leela.

dkappe
Posts: 235
Joined: Tue Aug 21, 2018 5:52 pm
Full name: Dietrich Kappe

Re: How to find your Leela Ratio - for Windows users w/ GPU(s)

Post by dkappe » Thu Feb 28, 2019 3:25 pm

If you want to return to the dark ages of “hey, leela just beat stockfish on my computer,” be my guest. If you have a better benchmark, let’s see it. It’s easy to throw stones.

User avatar
sincerelybrent
Posts: 47
Joined: Fri Feb 15, 2019 10:41 pm
Full name: Brent M

Re: How to find your Leela Ratio - for Windows users w/ GPU(s)

Post by sincerelybrent » Thu Feb 28, 2019 3:31 pm

Laskos wrote:
Thu Feb 28, 2019 7:19 am
... The so called effective "Leela Ratio" is different by a factor of 2 or more going from say 15''+0.25'' to 240''+4'', and maybe a factor of 3 or more compared to this "benchmark". The size of cache also matters...

This is still useful benchmark to determine the general hardware configuration GPU vs CPU in Leela case, but that's pretty much all.
You can change Leela Ratio settings to match conditions you would use for different time controls, as I show in the doc, if you’d like. Bullet vs TCEC time controls, for example. But the changes to nncache, like you mention, won’t affect the ratio much, unless you’re choking Leela by giving way too little. The biggest factor in changing the ratio’s value is backend, as I’m sure you’re aware. As long as that’s correct, you’ll be close. You mentioned a 10-15% margin in other test groups for CPU testing being acceptable. If we can get our GPU to CPU ratios within that margin, why not? That’s a lot better than a 1.5x - 3.8x difference in my rig.

Without some benchmark, two variables are in the test: GPU-to-CPU difference and the engines. If we can make the GPU to CPU ratio as close to 1:1 as we can, then we are testing the engines only, without there being a great disparity between GPU and CPU. Was A0 truly better? Is it that much better than Stockfish? We don’t know, because A0 had such an advantage in “time.” Set the ratio to 1:1 and have the match again. What would the result be?

How would you solve the problem of “leveling the playing field” for GPU vs CPU engines?

User avatar
sincerelybrent
Posts: 47
Joined: Fri Feb 15, 2019 10:41 pm
Full name: Brent M

Re: How to find your Leela Ratio - for Windows users w/ GPU(s)

Post by sincerelybrent » Thu Feb 28, 2019 3:34 pm

dkappe wrote:
Thu Feb 28, 2019 3:25 pm
If you want to return to the dark ages of “hey, leela just beat stockfish on my computer,” be my guest. If you have a better benchmark, let’s see it. It’s easy to throw stones.
Exactly.

As an example, when I set my time controls to make the Leela Ratio 1:1, Stockfish and Leela were dead even after 100 games. When I don’t adjust the time and just let Leela rip, she spanks Stockfish and scores almost quadruple the number of wins.

User avatar
Laskos
Posts: 9326
Joined: Wed Jul 26, 2006 8:21 pm
Full name: Kai Laskos

Re: How to find your Leela Ratio - for Windows users w/ GPU(s)

Post by Laskos » Thu Feb 28, 2019 4:53 pm

brianr wrote:
Thu Feb 28, 2019 1:15 pm
Laskos wrote:
Thu Feb 28, 2019 7:19 am
I stressed what I wanted to say because I often use very short time controls, similar to those in Fishtest testing framework, often one needs many games. The so called effective "Leela Ratio" is different by a factor of 2 or more going from say 15''+0.25'' to 240''+4'', and maybe a factor of 3 or more compared to this "benchmark". The size of cache also matters. With CPUs the speeds vary only by at most 10-15%% on these factors, not 300%, that's why CCRL benchmark is of much more use.

This is still useful benchmark to determine the general hardware configuration GPU vs CPU in Leela case, but that's pretty much all.
I have found that when running on a GPU there is significant startup overhead and the NPS increases very rapidly until about 2 seconds (and slowly continues to increase after that until the nncache fills). This is on a 1070. Accordingly, although the games take more time, I no longer test at faster time controls than with an increment of 2 seconds. I have seen match results completely reversed at faster TCs vs 1+2, even well within the margin of error. Leela NPS is also much more variable depending on the position than the NPS for CPU engines. Of course, CPUs have practically no overhead, so the ultra-rapid TC testing is fine. Moreover, the Leela Ratio only applies to nets with a specific size/architecture, as has been pointed out.

Just noticed more info in this post:
viewtopic.php?f=7&t=67347
Thanks for the link.

On RTX 2070 it is even worse than on 1070, from 2 seconds NPS still increases further to some maybe 30% higher at 20 seconds and maybe another 20% until the cache fills in to say 50%, which for large cache can be 200 seconds or more. At short times like 0.6s compared to 0.2s, the NPS could be a factor of 2 higher, the things are uncontrollable there as anything so called "Leela Ratio" goes due to overhead. Averaging over in-game positions at any TC works badly, LittleBlitzer does that, with stupid results. Suddenly on an in-game position, cache hits seem to dominate, giving some five fold sudden increase in speed, a thing never happening with CPU regular engines. Averaging with these spikes gives pretty meaningless results. Also the speed varies pretty significantly from net to net even inside the same run, say T40 run. Some of the nets there are say 20-30% slower than other using the same engine (same run, same net size, architecture, etc.)

But it's still good to have this benchmark in order to have some idea about hardware configuration. For now it's easy to form an idea, say you have a strong RTX GPU and a quad- or six-core CPU, means pretty high so called "Leela Ratio" in say Blitz or longer TC. But more GPUs will appear, from AMD too, possibly with tensor cores, and this benchmark will be somehow useful to give some impression.

mwyoung
Posts: 1608
Joined: Wed May 12, 2010 8:00 pm

Re: How to find your Leela Ratio - for Windows users w/ GPU(s)

Post by mwyoung » Thu Feb 28, 2019 7:25 pm

Mike S. wrote:
Thu Feb 28, 2019 1:44 pm
I think the whole concept of "Leela Ratio" is useless. You'll take your best buy anyway.
I agree it is meaningless. My ratio is 2950x vs 2080 ti.

All you need to know is gpu and settings. And as long as you test against known ratings of the other engines. You know the true strength of Lc0 on a given gpu.
Professing themselves to be wise, they became fools,
Take on me. foes 0

Post Reply