TalkChess.com

Hopefully this will save you a lot of time. I'd like to thank @dkappe for creating this tool and for his help w/ getting it set up on my computer. How to find your Leela Ratio

If you don't want to go through all of that, maybe this will give you a reference:

My GPU is GeForce RTX 2070
My CPUs are Intel i7-8700 @ 3.2 GHz

My Leela ratio with the cudnn-fp16 backend = 1.64 (1.64:1)
My Leelafish ratio with the cudnn-fp16 backend = 3.84

My Leela ratio with the (slower) cudnn backend = 0.66
My Leelafish ratio with the (slower) cudnn backend = 1.55

sincerelybrent wrote: ↑Wed Feb 27, 2019 11:08 pm Hopefully this will save you a lot of time. I'd like to thank @dkappe for creating this tool and for his help w/ getting it set up on my computer. How to find your Leela Ratio

If you don't want to go through all of that, maybe this will give you a reference:

My GPU is GeForce RTX 2070
My CPUs are Intel i7-8700 @ 3.2 GHz

My Leela ratio with the cudnn-fp16 backend = 1.64 (1.64:1)
My Leelafish ratio with the cudnn-fp16 backend = 3.84

My Leela ratio with the (slower) cudnn backend = 0.66
My Leelafish ratio with the (slower) cudnn backend = 1.55

Constant Leela Ratio is by a large misleading. It depends on time control and on type of positions used. For example, it is in the ballpark of 30% higher in 10' + 10'' games than in 1' + 1'' games (on RTX GPU and large NN cache).

Laskos wrote: ↑Thu Feb 28, 2019 12:10 am

sincerelybrent wrote: ↑Wed Feb 27, 2019 11:08 pm Hopefully this will save you a lot of time. I'd like to thank @dkappe for creating this tool and for his help w/ getting it set up on my computer. How to find your Leela Ratio

If you don't want to go through all of that, maybe this will give you a reference:

My GPU is GeForce RTX 2070
My CPUs are Intel i7-8700 @ 3.2 GHz

My Leela ratio with the cudnn-fp16 backend = 1.64 (1.64:1)
My Leelafish ratio with the cudnn-fp16 backend = 3.84

My Leela ratio with the (slower) cudnn backend = 0.66
My Leelafish ratio with the (slower) cudnn backend = 1.55

Constant Leela Ratio is by a large misleading. It depends on time control and on type of positions used. For example, it is in the ballpark of 30% higher in 10' + 10'' games than in 1' + 1'' games (on RTX GPU and large NN cache).

You misunderstand the purpose of the leela ratio. It is a measure of gpu vs cpu performance, not of chess performance. As such one should use sf9 and net 11248 (or 11258) from the start position. It's much like the crafty benchmark used by CCRL. Yes, that particular version of crafty will perform differently in different positions, but as a CPU benchmark, it works just fine.

dkappe wrote: ↑Thu Feb 28, 2019 1:10 am

Laskos wrote: ↑Thu Feb 28, 2019 12:10 am

sincerelybrent wrote: ↑Wed Feb 27, 2019 11:08 pm Hopefully this will save you a lot of time. I'd like to thank @dkappe for creating this tool and for his help w/ getting it set up on my computer. How to find your Leela Ratio

If you don't want to go through all of that, maybe this will give you a reference:

My GPU is GeForce RTX 2070
My CPUs are Intel i7-8700 @ 3.2 GHz

My Leela ratio with the cudnn-fp16 backend = 1.64 (1.64:1)
My Leelafish ratio with the cudnn-fp16 backend = 3.84

My Leela ratio with the (slower) cudnn backend = 0.66
My Leelafish ratio with the (slower) cudnn backend = 1.55

Constant Leela Ratio is by a large misleading. It depends on time control and on type of positions used. For example, it is in the ballpark of 30% higher in 10' + 10'' games than in 1' + 1'' games (on RTX GPU and large NN cache).

You misunderstand the purpose of the leela ratio. It is a measure of gpu vs cpu performance, not of chess performance. As such one should use sf9 and net 11248 (or 11258) from the start position. It's much like the crafty benchmark used by CCRL. Yes, that particular version of crafty will perform differently in different positions, but as a CPU benchmark, it works just fine.

Well, then it is just an arbitrary benchmark. But is is used many times when one shows the results of some tests. In that case this benchmark is not of much use if he doesn't take into account all other factors.
CCRL benchmark is much more reliable one for CPU engines, the variations using different time controls, hash sizes etc. are less than 10%.

Laskos wrote: ↑Thu Feb 28, 2019 6:04 am
Well, then it is just an arbitrary benchmark. But is is used many times when one shows the results of some tests. In that case this benchmark is not of much use if he doesn't take into account all other factors.
CCRL benchmark is much more reliable one for CPU engines, the variations using different time controls, hash sizes etc. are less than 10%.

It is semi-arbitrary (based on the reported nps ratio between a0 and sf8), yet predictive of performance at various tc and positions. I assume you want a much more complex benchmark? I’m eager to see it.

dkappe wrote: ↑Thu Feb 28, 2019 7:09 am

Laskos wrote: ↑Thu Feb 28, 2019 6:04 am
Well, then it is just an arbitrary benchmark. But is is used many times when one shows the results of some tests. In that case this benchmark is not of much use if he doesn't take into account all other factors.
CCRL benchmark is much more reliable one for CPU engines, the variations using different time controls, hash sizes etc. are less than 10%.

It is semi-arbitrary (based on the reported nps ratio between a0 and sf8), yet predictive of performance at various tc and positions. I assume you want a much more complex benchmark? I’m eager to see it.

I stressed what I wanted to say because I often use very short time controls, similar to those in Fishtest testing framework, often one needs many games. The so called effective "Leela Ratio" is different by a factor of 2 or more going from say 15''+0.25'' to 240''+4'', and maybe a factor of 3 or more compared to this "benchmark". The size of cache also matters. With CPUs the speeds vary only by at most 10-15%% on these factors, not 300%, that's why CCRL benchmark is of much more use.

This is still useful benchmark to determine the general hardware configuration GPU vs CPU in Leela case, but that's pretty much all.

sincerelybrent wrote: ↑Wed Feb 27, 2019 11:08 pm Hopefully this will save you a lot of time. I'd like to thank @dkappe for creating this tool and for his help w/ getting it set up on my computer. How to find your Leela Ratio

If you don't want to go through all of that, maybe this will give you a reference:

My GPU is GeForce RTX 2070
My CPUs are Intel i7-8700 @ 3.2 GHz

My Leela ratio with the cudnn-fp16 backend = 1.64 (1.64:1)
My Leelafish ratio with the cudnn-fp16 backend = 3.84

My Leela ratio with the (slower) cudnn backend = 0.66
My Leelafish ratio with the (slower) cudnn backend = 1.55

I think Leela Ratio is very useful to get an idea of ​​the relative speed of the GPU and the CPU.

But I don't understand why the Leela Ratio grows with LeelaFish.
Is taking into account the nodes of the auxiliary engine?
How?

Javier Ros wrote: ↑Thu Feb 28, 2019 9:35 am

sincerelybrent wrote: ↑Wed Feb 27, 2019 11:08 pm Hopefully this will save you a lot of time. I'd like to thank @dkappe for creating this tool and for his help w/ getting it set up on my computer. How to find your Leela Ratio

If you don't want to go through all of that, maybe this will give you a reference:

My GPU is GeForce RTX 2070
My CPUs are Intel i7-8700 @ 3.2 GHz

My Leela ratio with the cudnn-fp16 backend = 1.64 (1.64:1)
My Leelafish ratio with the cudnn-fp16 backend = 3.84

My Leela ratio with the (slower) cudnn backend = 0.66
My Leelafish ratio with the (slower) cudnn backend = 1.55

I think Leela Ratio is very useful to get an idea of ​​the relative speed of the GPU and the CPU.

But I don't understand why the Leela Ratio grows with LeelaFish.
Is taking into account the nodes of the auxiliary engine?
How?

You shouldn’t use leelafish to compute the Leela Ratio. Use lc0 with 11248 vs sf9.

Laskos wrote: ↑Thu Feb 28, 2019 8:19 am I stressed what I wanted to say because I often use very short time controls, similar to those in Fishtest testing framework, often one needs many games. The so called effective "Leela Ratio" is different by a factor of 2 or more going from say 15''+0.25'' to 240''+4'', and maybe a factor of 3 or more compared to this "benchmark". The size of cache also matters. With CPUs the speeds vary only by at most 10-15%% on these factors, not 300%, that's why CCRL benchmark is of much more use.

This is still useful benchmark to determine the general hardware configuration GPU vs CPU in Leela case, but that's pretty much all.

I have found that when running on a GPU there is significant startup overhead and the NPS increases very rapidly until about 2 seconds (and slowly continues to increase after that until the nncache fills). This is on a 1070. Accordingly, although the games take more time, I no longer test at faster time controls than with an increment of 2 seconds. I have seen match results completely reversed at faster TCs vs 1+2, even well within the margin of error. Leela NPS is also much more variable depending on the position than the NPS for CPU engines. Of course, CPUs have practically no overhead, so the ultra-rapid TC testing is fine. Moreover, the Leela Ratio only applies to nets with a specific size/architecture, as has been pointed out.

Just noticed more info in this post:
viewtopic.php?f=7&t=67347

I think the whole concept of "Leela Ratio" is useless. You'll take your best buy anyway.