How do you know if your cpu is equal to your gpu?

[dkappe]

You asked a controversial question: when are cpu and gpu “equal?” At best it’s difficult if not impossible to answer. How do you define “equal?”

A much more useful question would be “how do I know if matches run on two different machines between gpu and cpu engines are comparable?” A primitive approach has been to determine the nps ratio from the start position for sf8 and 11248 running on lc0 v18. The hypothesis is that if those ratios are the same, then the results would be comparable.

Not everyone has those engines available to them, so it might be both primitive and impractical.

[jp]

The hypothesis is that if those ratios are the same, then the results would be comparable.

That of course depends on the curves (performance vs. time or nodes per move) being the same shape, so again there's need for those curves.

And comparable results can still be dangerous, because invariably people will think of them as "comparably fair", not "equally unfair".

[dkappe]

That of course depends on the curves (performance vs. time or nodes per move) being the same shape, so again there's need for those curves.

There was an attempt to create a better “leela ratio” maybe a month ago. It turned out not to be successful. Instead they decided to peg a ratio at that of the current TCEC hardware (which will famously change this season). I’ve tested the ratio hypothesis — that the same ratio on different hardware leads to similar results — on a 1060, a 1070 and a 2070. I’d need many more thousands of games, but I think it’s good enough to upgrade it from a hypothesis to a “thesis.”

And comparable results can still be dangerous, because invariably people will think of them as "comparably fair", not "equally unfair".

Meh. Now you’re getting into the region of religion. The point may be moot when CPU’s begin to pick up the characteristics of GPU’s.

[jp]

There was an attempt to create a better “leela ratio” maybe a month ago. It turned out not to be successful.

In what way was it unsuccessful?

Why hasn't there been more interest in Cscuile's ratio? Obviously it takes some effort to find (if it changes, and if it doesn't you could say that means no real progress), but where an engine goes flatline is what actually matters, not hardware.

Identify the points at which flatlining starts and use that ratio. That way, it's independent of hardware capabilities, etc.

https://groups.google.com/forum/#!msg/l ... oJrjDgBQAJ

[dkappe]

There was an attempt to create a better “leela ratio” maybe a month ago. It turned out not to be successful.

In what way was it unsuccessful?

Too complex and unworkable vs a simple benchmark.

Why hasn't there been more interest in Cscuile's ratio? Obviously it takes some effort to find (if it changes, and if it doesn't you could say that means no real progress), but where an engine goes flatline is what actually matters, not hardware.

Probably for the same reasons -- too complex and unworkable. Also, the concept it not fully fleshed out and there's no evidence to far that its useful for comparing results across hardware.

[jp]

Also, the concept it not fully fleshed out and there's no evidence to far that its useful for comparing results across hardware.

If one ratio "works" (i.e. allows comparison) across different time controls and hardware, as you suggested before, then another more meaningful ratio should also work.


Leela maxing out its performance at 200000 npm in Cscuile's testing back then is not good.
Testing that just avoids this issue is also not good.

[dkappe]

If one ratio "works" (i.e. allows comparison) across different time controls and hardware, as you suggested before, then another more meaningful ratio should also work.

More meaningful? Ratio = C * leela nps / sf nps. The only thing that changed was the C. Meh.

[jp]

Yes, more meaningful. Cscuile's ratio is not arbitrary and has something to do with chess and computer chess. The other ratio is arbitrary and has almost nothing to do with chess or computer chess.

Leela maxing out its performance at 200000 npm in Cscuile's testing back then is not good.
Testing that just avoids this issue is also not good.

[dkappe]

Well, I’m glad you have a favorite ratio. Cscuile has had so many ratios with so many formulas and acronyms that I couldn’t keep track of them all. But his heart was in the right place.

[jp]

I don't know what all Cscuile's ideas were, so I can't judge them all, just this one.
Belittling him won't change the merits of this ratio, nor will it increase the merits of a largely meaningless one.

What matters most is the (shape of the) curves of performance vs. computation time.
The question is whether Leela (or other engines) still hits the wall at 200000npm.