Let me see if I understand what you are saying. We can consider one "heat" to be a small match between engine 1 and engine 2 where we continue playing games until we get a result that is not a draw. There is a true probability of engine 1 winning the heat, and we call it p.Laskos wrote:Bayes' formula to derive LOS uses the uniform prior (and thus giving that nice closed form and erf approximation). I used the same Bayes' formula to derive LOSp (non-uniform prior) with numerical results.AlvaroBegue wrote:I don't understand what you are saying. LOS as a p-value doesn't use a prior at all, uniform or otherwise. It is a measure of statistical significance of a departure from the null hypothesis. There is really nothing wrong with it.Laskos wrote:LOS as p-value is indeed defined with an uniform prior. So, a pretty bad quantity to use for us.AlvaroBegue wrote:I don't think of LOS in any Bayesian framework: LOS is what in other fields is called a p-value. It's a quantity that, under the null hypothesis that both players are equally strong (i.e., if the true Elo difference is 0), would be uniformly distributed in [0,1].

See https://en.wikipedia.org/wiki/P-value .

We could discover something about p by using Bayesian statistics, where we start with some prior, we observe some results of heats and we then get a posterior probability. We might be interested in answering questions like "what is the probability that p is larger than 0.5?".

If we use a uniform prior, that probability is the LOS as it's usually defined. If we use a different prior (I think you suggest a Beta(1001,1001) distribution), we'll get an alternative definition (which will look a lot like assuming an initial tally of 1000 wins and 1000 losses).

Are we together so far?

What I am saying is that you can define LOS as a p-value of the results, which is a test of the plausibility of the null hypothesis. This is a frequentist approach to the problem, and not a Bayesian one. This is how I think of the meaning of LOS, and nothing else. It's still a very useful number, but it needs to be interpreted carefully, just like any p-value.

WHAAAT??CPW wrote:The likelihood of superiority (LOS) denotes the probability of a certain engine being stronger than another.

Now I see what your beef is about! Your Bayesian interpretation of LOS with uniform prior would give some meaning to that sentence, but assuming a uniform prior is unreasonable. The other possibility is that whoever wrote that is being tripped by a very common misunderstanding of p-values. So common in fact that it has its own Wikipedia page: https://en.wikipedia.org/wiki/Misunders ... f_p-values