How to find your Leela Ratio - for Windows users w/ GPU(s)

Discussion of anything and everything relating to chess playing software and machines.

Moderators: Harvey Williamson, Dann Corbit, hgm

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
sincerelybrent
Posts: 47
Joined: Fri Feb 15, 2019 10:41 pm
Full name: Brent M

How to find your Leela Ratio - for Windows users w/ GPU(s)

Hopefully this will save you a lot of time. I'd like to thank @dkappe for creating this tool and for his help w/ getting it set up on my computer. How to find your Leela Ratio

If you don't want to go through all of that, maybe this will give you a reference:

My GPU is GeForce RTX 2070
My CPUs are Intel i7-8700 @ 3.2 GHz

My Leela ratio with the cudnn-fp16 backend = 1.64 (1.64:1)
My Leelafish ratio with the cudnn-fp16 backend = 3.84

My Leela ratio with the (slower) cudnn backend = 0.66
My Leelafish ratio with the (slower) cudnn backend = 1.55

Posts: 10945
Joined: Wed Jul 26, 2006 8:21 pm

Re: How to find your Leela Ratio - for Windows users w/ GPU(s)

sincerelybrent wrote:
Wed Feb 27, 2019 10:08 pm
Hopefully this will save you a lot of time. I'd like to thank @dkappe for creating this tool and for his help w/ getting it set up on my computer. How to find your Leela Ratio

If you don't want to go through all of that, maybe this will give you a reference:

My GPU is GeForce RTX 2070
My CPUs are Intel i7-8700 @ 3.2 GHz

My Leela ratio with the cudnn-fp16 backend = 1.64 (1.64:1)
My Leelafish ratio with the cudnn-fp16 backend = 3.84

My Leela ratio with the (slower) cudnn backend = 0.66
My Leelafish ratio with the (slower) cudnn backend = 1.55
Constant Leela Ratio is by a large misleading. It depends on time control and on type of positions used. For example, it is in the ballpark of 30% higher in 10' + 10'' games than in 1' + 1'' games (on RTX GPU and large NN cache).

dkappe
Posts: 574
Joined: Tue Aug 21, 2018 5:52 pm
Full name: Dietrich Kappe

Re: How to find your Leela Ratio - for Windows users w/ GPU(s)

Wed Feb 27, 2019 11:10 pm
sincerelybrent wrote:
Wed Feb 27, 2019 10:08 pm
Hopefully this will save you a lot of time. I'd like to thank @dkappe for creating this tool and for his help w/ getting it set up on my computer. How to find your Leela Ratio

If you don't want to go through all of that, maybe this will give you a reference:

My GPU is GeForce RTX 2070
My CPUs are Intel i7-8700 @ 3.2 GHz

My Leela ratio with the cudnn-fp16 backend = 1.64 (1.64:1)
My Leelafish ratio with the cudnn-fp16 backend = 3.84

My Leela ratio with the (slower) cudnn backend = 0.66
My Leelafish ratio with the (slower) cudnn backend = 1.55
Constant Leela Ratio is by a large misleading. It depends on time control and on type of positions used. For example, it is in the ballpark of 30% higher in 10' + 10'' games than in 1' + 1'' games (on RTX GPU and large NN cache).
You misunderstand the purpose of the leela ratio. It is a measure of gpu vs cpu performance, not of chess performance. As such one should use sf9 and net 11248 (or 11258) from the start position. It's much like the crafty benchmark used by CCRL. Yes, that particular version of crafty will perform differently in different positions, but as a CPU benchmark, it works just fine.

Posts: 10945
Joined: Wed Jul 26, 2006 8:21 pm

Re: How to find your Leela Ratio - for Windows users w/ GPU(s)

dkappe wrote:
Thu Feb 28, 2019 12:10 am
Wed Feb 27, 2019 11:10 pm
sincerelybrent wrote:
Wed Feb 27, 2019 10:08 pm
Hopefully this will save you a lot of time. I'd like to thank @dkappe for creating this tool and for his help w/ getting it set up on my computer. How to find your Leela Ratio

If you don't want to go through all of that, maybe this will give you a reference:

My GPU is GeForce RTX 2070
My CPUs are Intel i7-8700 @ 3.2 GHz

My Leela ratio with the cudnn-fp16 backend = 1.64 (1.64:1)
My Leelafish ratio with the cudnn-fp16 backend = 3.84

My Leela ratio with the (slower) cudnn backend = 0.66
My Leelafish ratio with the (slower) cudnn backend = 1.55
Constant Leela Ratio is by a large misleading. It depends on time control and on type of positions used. For example, it is in the ballpark of 30% higher in 10' + 10'' games than in 1' + 1'' games (on RTX GPU and large NN cache).
You misunderstand the purpose of the leela ratio. It is a measure of gpu vs cpu performance, not of chess performance. As such one should use sf9 and net 11248 (or 11258) from the start position. It's much like the crafty benchmark used by CCRL. Yes, that particular version of crafty will perform differently in different positions, but as a CPU benchmark, it works just fine.
Well, then it is just an arbitrary benchmark. But is is used many times when one shows the results of some tests. In that case this benchmark is not of much use if he doesn't take into account all other factors.
CCRL benchmark is much more reliable one for CPU engines, the variations using different time controls, hash sizes etc. are less than 10%.

dkappe
Posts: 574
Joined: Tue Aug 21, 2018 5:52 pm
Full name: Dietrich Kappe

Re: How to find your Leela Ratio - for Windows users w/ GPU(s)

Thu Feb 28, 2019 5:04 am

Well, then it is just an arbitrary benchmark. But is is used many times when one shows the results of some tests. In that case this benchmark is not of much use if he doesn't take into account all other factors.
CCRL benchmark is much more reliable one for CPU engines, the variations using different time controls, hash sizes etc. are less than 10%.
It is semi-arbitrary (based on the reported nps ratio between a0 and sf8), yet predictive of performance at various tc and positions. I assume you want a much more complex benchmark? I’m eager to see it.

Posts: 10945
Joined: Wed Jul 26, 2006 8:21 pm

Re: How to find your Leela Ratio - for Windows users w/ GPU(s)

dkappe wrote:
Thu Feb 28, 2019 6:09 am
Thu Feb 28, 2019 5:04 am

Well, then it is just an arbitrary benchmark. But is is used many times when one shows the results of some tests. In that case this benchmark is not of much use if he doesn't take into account all other factors.
CCRL benchmark is much more reliable one for CPU engines, the variations using different time controls, hash sizes etc. are less than 10%.
It is semi-arbitrary (based on the reported nps ratio between a0 and sf8), yet predictive of performance at various tc and positions. I assume you want a much more complex benchmark? I’m eager to see it.
I stressed what I wanted to say because I often use very short time controls, similar to those in Fishtest testing framework, often one needs many games. The so called effective "Leela Ratio" is different by a factor of 2 or more going from say 15''+0.25'' to 240''+4'', and maybe a factor of 3 or more compared to this "benchmark". The size of cache also matters. With CPUs the speeds vary only by at most 10-15%% on these factors, not 300%, that's why CCRL benchmark is of much more use.

This is still useful benchmark to determine the general hardware configuration GPU vs CPU in Leela case, but that's pretty much all.

Javier Ros
Posts: 184
Joined: Fri Oct 12, 2012 10:48 am
Location: Seville (SPAIN)
Full name: Javier Ros

Re: How to find your Leela Ratio - for Windows users w/ GPU(s)

sincerelybrent wrote:
Wed Feb 27, 2019 10:08 pm
Hopefully this will save you a lot of time. I'd like to thank @dkappe for creating this tool and for his help w/ getting it set up on my computer. How to find your Leela Ratio

If you don't want to go through all of that, maybe this will give you a reference:

My GPU is GeForce RTX 2070
My CPUs are Intel i7-8700 @ 3.2 GHz

My Leela ratio with the cudnn-fp16 backend = 1.64 (1.64:1)
My Leelafish ratio with the cudnn-fp16 backend = 3.84

My Leela ratio with the (slower) cudnn backend = 0.66
My Leelafish ratio with the (slower) cudnn backend = 1.55
I think Leela Ratio is very useful to get an idea of ​​the relative speed of the GPU and the CPU.

But I don't understand why the Leela Ratio grows with LeelaFish.
Is taking into account the nodes of the auxiliary engine?
How?
The love relationship between a chess engine tester and his computer can be summarized in one sentence:
Until heat do us part.

dkappe
Posts: 574
Joined: Tue Aug 21, 2018 5:52 pm
Full name: Dietrich Kappe

Re: How to find your Leela Ratio - for Windows users w/ GPU(s)

Javier Ros wrote:
Thu Feb 28, 2019 8:35 am
sincerelybrent wrote:
Wed Feb 27, 2019 10:08 pm
Hopefully this will save you a lot of time. I'd like to thank @dkappe for creating this tool and for his help w/ getting it set up on my computer. How to find your Leela Ratio

If you don't want to go through all of that, maybe this will give you a reference:

My GPU is GeForce RTX 2070
My CPUs are Intel i7-8700 @ 3.2 GHz

My Leela ratio with the cudnn-fp16 backend = 1.64 (1.64:1)
My Leelafish ratio with the cudnn-fp16 backend = 3.84

My Leela ratio with the (slower) cudnn backend = 0.66
My Leelafish ratio with the (slower) cudnn backend = 1.55
I think Leela Ratio is very useful to get an idea of ​​the relative speed of the GPU and the CPU.

But I don't understand why the Leela Ratio grows with LeelaFish.
Is taking into account the nodes of the auxiliary engine?
How?
You shouldn’t use leelafish to compute the Leela Ratio. Use lc0 with 11248 vs sf9.

brianr
Posts: 457
Joined: Thu Mar 09, 2006 2:01 pm

Re: How to find your Leela Ratio - for Windows users w/ GPU(s)

Thu Feb 28, 2019 7:19 am
I stressed what I wanted to say because I often use very short time controls, similar to those in Fishtest testing framework, often one needs many games. The so called effective "Leela Ratio" is different by a factor of 2 or more going from say 15''+0.25'' to 240''+4'', and maybe a factor of 3 or more compared to this "benchmark". The size of cache also matters. With CPUs the speeds vary only by at most 10-15%% on these factors, not 300%, that's why CCRL benchmark is of much more use.

This is still useful benchmark to determine the general hardware configuration GPU vs CPU in Leela case, but that's pretty much all.
I have found that when running on a GPU there is significant startup overhead and the NPS increases very rapidly until about 2 seconds (and slowly continues to increase after that until the nncache fills). This is on a 1070. Accordingly, although the games take more time, I no longer test at faster time controls than with an increment of 2 seconds. I have seen match results completely reversed at faster TCs vs 1+2, even well within the margin of error. Leela NPS is also much more variable depending on the position than the NPS for CPU engines. Of course, CPUs have practically no overhead, so the ultra-rapid TC testing is fine. Moreover, the Leela Ratio only applies to nets with a specific size/architecture, as has been pointed out.