How to find your Leela Ratio - for Windows users w/ GPU(s)

Discussion of anything and everything relating to chess playing software and machines.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
User avatar
sincerelybrent
Posts: 47
Joined: Fri Feb 15, 2019 10:41 pm
Full name: Brent M

How to find your Leela Ratio - for Windows users w/ GPU(s)

Post by sincerelybrent » Wed Feb 27, 2019 10:08 pm

Hopefully this will save you a lot of time. I'd like to thank @dkappe for creating this tool and for his help w/ getting it set up on my computer. How to find your Leela Ratio

If you don't want to go through all of that, maybe this will give you a reference:

My GPU is GeForce RTX 2070
My CPUs are Intel i7-8700 @ 3.2 GHz

My Leela ratio with the cudnn-fp16 backend = 1.64 (1.64:1)
My Leelafish ratio with the cudnn-fp16 backend = 3.84

My Leela ratio with the (slower) cudnn backend = 0.66
My Leelafish ratio with the (slower) cudnn backend = 1.55

User avatar
Laskos
Posts: 9410
Joined: Wed Jul 26, 2006 8:21 pm
Full name: Kai Laskos

Re: How to find your Leela Ratio - for Windows users w/ GPU(s)

Post by Laskos » Wed Feb 27, 2019 11:10 pm

sincerelybrent wrote:
Wed Feb 27, 2019 10:08 pm
Hopefully this will save you a lot of time. I'd like to thank @dkappe for creating this tool and for his help w/ getting it set up on my computer. How to find your Leela Ratio

If you don't want to go through all of that, maybe this will give you a reference:

My GPU is GeForce RTX 2070
My CPUs are Intel i7-8700 @ 3.2 GHz

My Leela ratio with the cudnn-fp16 backend = 1.64 (1.64:1)
My Leelafish ratio with the cudnn-fp16 backend = 3.84

My Leela ratio with the (slower) cudnn backend = 0.66
My Leelafish ratio with the (slower) cudnn backend = 1.55
Constant Leela Ratio is by a large misleading. It depends on time control and on type of positions used. For example, it is in the ballpark of 30% higher in 10' + 10'' games than in 1' + 1'' games (on RTX GPU and large NN cache).

dkappe
Posts: 235
Joined: Tue Aug 21, 2018 5:52 pm
Full name: Dietrich Kappe

Re: How to find your Leela Ratio - for Windows users w/ GPU(s)

Post by dkappe » Thu Feb 28, 2019 12:10 am

Laskos wrote:
Wed Feb 27, 2019 11:10 pm
sincerelybrent wrote:
Wed Feb 27, 2019 10:08 pm
Hopefully this will save you a lot of time. I'd like to thank @dkappe for creating this tool and for his help w/ getting it set up on my computer. How to find your Leela Ratio

If you don't want to go through all of that, maybe this will give you a reference:

My GPU is GeForce RTX 2070
My CPUs are Intel i7-8700 @ 3.2 GHz

My Leela ratio with the cudnn-fp16 backend = 1.64 (1.64:1)
My Leelafish ratio with the cudnn-fp16 backend = 3.84

My Leela ratio with the (slower) cudnn backend = 0.66
My Leelafish ratio with the (slower) cudnn backend = 1.55
Constant Leela Ratio is by a large misleading. It depends on time control and on type of positions used. For example, it is in the ballpark of 30% higher in 10' + 10'' games than in 1' + 1'' games (on RTX GPU and large NN cache).
You misunderstand the purpose of the leela ratio. It is a measure of gpu vs cpu performance, not of chess performance. As such one should use sf9 and net 11248 (or 11258) from the start position. It's much like the crafty benchmark used by CCRL. Yes, that particular version of crafty will perform differently in different positions, but as a CPU benchmark, it works just fine.

User avatar
Laskos
Posts: 9410
Joined: Wed Jul 26, 2006 8:21 pm
Full name: Kai Laskos

Re: How to find your Leela Ratio - for Windows users w/ GPU(s)

Post by Laskos » Thu Feb 28, 2019 5:04 am

dkappe wrote:
Thu Feb 28, 2019 12:10 am
Laskos wrote:
Wed Feb 27, 2019 11:10 pm
sincerelybrent wrote:
Wed Feb 27, 2019 10:08 pm
Hopefully this will save you a lot of time. I'd like to thank @dkappe for creating this tool and for his help w/ getting it set up on my computer. How to find your Leela Ratio

If you don't want to go through all of that, maybe this will give you a reference:

My GPU is GeForce RTX 2070
My CPUs are Intel i7-8700 @ 3.2 GHz

My Leela ratio with the cudnn-fp16 backend = 1.64 (1.64:1)
My Leelafish ratio with the cudnn-fp16 backend = 3.84

My Leela ratio with the (slower) cudnn backend = 0.66
My Leelafish ratio with the (slower) cudnn backend = 1.55
Constant Leela Ratio is by a large misleading. It depends on time control and on type of positions used. For example, it is in the ballpark of 30% higher in 10' + 10'' games than in 1' + 1'' games (on RTX GPU and large NN cache).
You misunderstand the purpose of the leela ratio. It is a measure of gpu vs cpu performance, not of chess performance. As such one should use sf9 and net 11248 (or 11258) from the start position. It's much like the crafty benchmark used by CCRL. Yes, that particular version of crafty will perform differently in different positions, but as a CPU benchmark, it works just fine.
Well, then it is just an arbitrary benchmark. But is is used many times when one shows the results of some tests. In that case this benchmark is not of much use if he doesn't take into account all other factors.
CCRL benchmark is much more reliable one for CPU engines, the variations using different time controls, hash sizes etc. are less than 10%.

dkappe
Posts: 235
Joined: Tue Aug 21, 2018 5:52 pm
Full name: Dietrich Kappe

Re: How to find your Leela Ratio - for Windows users w/ GPU(s)

Post by dkappe » Thu Feb 28, 2019 6:09 am

Laskos wrote:
Thu Feb 28, 2019 5:04 am

Well, then it is just an arbitrary benchmark. But is is used many times when one shows the results of some tests. In that case this benchmark is not of much use if he doesn't take into account all other factors.
CCRL benchmark is much more reliable one for CPU engines, the variations using different time controls, hash sizes etc. are less than 10%.
It is semi-arbitrary (based on the reported nps ratio between a0 and sf8), yet predictive of performance at various tc and positions. I assume you want a much more complex benchmark? I’m eager to see it.

User avatar
Laskos
Posts: 9410
Joined: Wed Jul 26, 2006 8:21 pm
Full name: Kai Laskos

Re: How to find your Leela Ratio - for Windows users w/ GPU(s)

Post by Laskos » Thu Feb 28, 2019 7:19 am

dkappe wrote:
Thu Feb 28, 2019 6:09 am
Laskos wrote:
Thu Feb 28, 2019 5:04 am

Well, then it is just an arbitrary benchmark. But is is used many times when one shows the results of some tests. In that case this benchmark is not of much use if he doesn't take into account all other factors.
CCRL benchmark is much more reliable one for CPU engines, the variations using different time controls, hash sizes etc. are less than 10%.
It is semi-arbitrary (based on the reported nps ratio between a0 and sf8), yet predictive of performance at various tc and positions. I assume you want a much more complex benchmark? I’m eager to see it.
I stressed what I wanted to say because I often use very short time controls, similar to those in Fishtest testing framework, often one needs many games. The so called effective "Leela Ratio" is different by a factor of 2 or more going from say 15''+0.25'' to 240''+4'', and maybe a factor of 3 or more compared to this "benchmark". The size of cache also matters. With CPUs the speeds vary only by at most 10-15%% on these factors, not 300%, that's why CCRL benchmark is of much more use.

This is still useful benchmark to determine the general hardware configuration GPU vs CPU in Leela case, but that's pretty much all.

Javier Ros
Posts: 181
Joined: Fri Oct 12, 2012 10:48 am
Location: Seville (SPAIN)
Full name: Javier Ros

Re: How to find your Leela Ratio - for Windows users w/ GPU(s)

Post by Javier Ros » Thu Feb 28, 2019 8:35 am

sincerelybrent wrote:
Wed Feb 27, 2019 10:08 pm
Hopefully this will save you a lot of time. I'd like to thank @dkappe for creating this tool and for his help w/ getting it set up on my computer. How to find your Leela Ratio

If you don't want to go through all of that, maybe this will give you a reference:

My GPU is GeForce RTX 2070
My CPUs are Intel i7-8700 @ 3.2 GHz

My Leela ratio with the cudnn-fp16 backend = 1.64 (1.64:1)
My Leelafish ratio with the cudnn-fp16 backend = 3.84

My Leela ratio with the (slower) cudnn backend = 0.66
My Leelafish ratio with the (slower) cudnn backend = 1.55
I think Leela Ratio is very useful to get an idea of ​​the relative speed of the GPU and the CPU.

But I don't understand why the Leela Ratio grows with LeelaFish.
Is taking into account the nodes of the auxiliary engine?
How?
The love relationship between a chess engine tester and his computer can be summarized in one sentence:
Until heat do us part.

dkappe
Posts: 235
Joined: Tue Aug 21, 2018 5:52 pm
Full name: Dietrich Kappe

Re: How to find your Leela Ratio - for Windows users w/ GPU(s)

Post by dkappe » Thu Feb 28, 2019 12:57 pm

Javier Ros wrote:
Thu Feb 28, 2019 8:35 am
sincerelybrent wrote:
Wed Feb 27, 2019 10:08 pm
Hopefully this will save you a lot of time. I'd like to thank @dkappe for creating this tool and for his help w/ getting it set up on my computer. How to find your Leela Ratio

If you don't want to go through all of that, maybe this will give you a reference:

My GPU is GeForce RTX 2070
My CPUs are Intel i7-8700 @ 3.2 GHz

My Leela ratio with the cudnn-fp16 backend = 1.64 (1.64:1)
My Leelafish ratio with the cudnn-fp16 backend = 3.84

My Leela ratio with the (slower) cudnn backend = 0.66
My Leelafish ratio with the (slower) cudnn backend = 1.55
I think Leela Ratio is very useful to get an idea of ​​the relative speed of the GPU and the CPU.

But I don't understand why the Leela Ratio grows with LeelaFish.
Is taking into account the nodes of the auxiliary engine?
How?
You shouldn’t use leelafish to compute the Leela Ratio. Use lc0 with 11248 vs sf9.

brianr
Posts: 356
Joined: Thu Mar 09, 2006 2:01 pm

Re: How to find your Leela Ratio - for Windows users w/ GPU(s)

Post by brianr » Thu Feb 28, 2019 1:15 pm

Laskos wrote:
Thu Feb 28, 2019 7:19 am
I stressed what I wanted to say because I often use very short time controls, similar to those in Fishtest testing framework, often one needs many games. The so called effective "Leela Ratio" is different by a factor of 2 or more going from say 15''+0.25'' to 240''+4'', and maybe a factor of 3 or more compared to this "benchmark". The size of cache also matters. With CPUs the speeds vary only by at most 10-15%% on these factors, not 300%, that's why CCRL benchmark is of much more use.

This is still useful benchmark to determine the general hardware configuration GPU vs CPU in Leela case, but that's pretty much all.
I have found that when running on a GPU there is significant startup overhead and the NPS increases very rapidly until about 2 seconds (and slowly continues to increase after that until the nncache fills). This is on a 1070. Accordingly, although the games take more time, I no longer test at faster time controls than with an increment of 2 seconds. I have seen match results completely reversed at faster TCs vs 1+2, even well within the margin of error. Leela NPS is also much more variable depending on the position than the NPS for CPU engines. Of course, CPUs have practically no overhead, so the ultra-rapid TC testing is fine. Moreover, the Leela Ratio only applies to nets with a specific size/architecture, as has been pointed out.

Just noticed more info in this post:
viewtopic.php?f=7&t=67347

User avatar
Mike S.
Posts: 1480
Joined: Thu Mar 09, 2006 4:33 am

Re: How to find your Leela Ratio - for Windows users w/ GPU(s)

Post by Mike S. » Thu Feb 28, 2019 1:44 pm

I think the whole concept of "Leela Ratio" is useless. You'll take your best buy anyway.
Regards, Mike

Post Reply