Likelihood Of Success (LOS) in the real world

[Dann Corbit]

LOS (Likelihood of superiority) can be interpreted as:
What are the chances that A is better than B?
If it is 50%, then a coin toss.
If it is 0% then no chance.
If it is 100% then certain.

Simple as that.

[Michel]

The Bayesian "probabilities" are not true probabilities. Instead they represent your "belief" that something is true. The prior is your original belief and as new information comes in you adapt your position. A uniform prior is a so-called non-informative prior. It is what you choose if you know nothing about a subject.

Interpreted in this way Bayesian statistics is suitable for reasoning but unsuitable for making precise scientific statements.

By some fortunate accident, in the case of fixed length tests, the LOS for a uniform prior happens to be almost the same as the p-value which has a precise scientific interpretation.

This equality is no longer true in the case of sequential tests and in that case it is easy to see that the naive interpretation of LOS as a probability leads to disastrous results.

[Laskos]

The Bayesian "probabilities" are not true probabilities. Instead they represent your "belief" that something is true. The prior is your original belief and as new information comes in you adapt your position. A uniform prior is a so-called non-informative prior. It is what you choose if you know nothing about a subject.

Interpreted in this way Bayesian statistics is suitable for reasoning but unsuitable for making precise scientific statements.

By some fortunate accident, in the case of fixed length tests, the LOS for a uniform prior happens to be almost the same as the p-value which has a precise scientific interpretation.

This equality is no longer true in the case of sequential tests and in that case it is easy to see that the naive interpretation of LOS as a probability leads to disastrous results.

P-value is often misused in scientific literature. Ask a physicist why 5-sigma is required. He will talk of probabilities, and will not be able to explain why such a high "confidence" as t=5 is required. P-value is dealing with Null hypothesis, and that's more rigorous but less informative than probability based on belief. Also, as reasoning goes, isn't the belief in uniform prior not only also a belief, but also a misplaced belief? It's hard to me to reason about anything having uniform prior. We take two Stockfishes, see 10-0-0 result and conclude that it has p-value of <0.001, test passed? What would be here a precise scientific statement? Practically, the belief in correct interpretation of p-value is more dangerous than belief using a prior. That 10-0-0 is 60% probability that 10 score engine is stronger, given the reasonable prior for Stockfishes.

[AlvaroBegue]

The Bayesian "probabilities" are not true probabilities. Instead they represent your "belief" that something is true.

This is an old debate which mathematicians no longer have. If you have a set X and a function P that maps certain subsets of X to the interval [0,1] and certain axioms are satisfied, we call P a probability. That's a definition. Bayesian probabilities are probabilities, and the whole theory of probability applies to them.

I am generally a big fan of a Bayesian approach to modeling uncertainty. However, coming up with a reasonable prior in some situations (like this one) is tricky, and the prior can have a large influence in the result of the analysis, as Kai points out. So I would rather use a frequentist approach here.

In order to use a p-value rigorously, you need to design the experiment in advance (i.e., decide how many games you are going to play or -even better- how many non-draw results you need, and decide what threshold of LOS you are going to use to accept or reject the new version). For instance, if I say I will play 10,000 games and accept the change if the LOS is above 0.995, I know that the probability of a non-improvement getting through is at most 0.005. If in my process of trying changes I have an acceptance rate much higher than 0.005 (say, 10%), I can be pretty certain that the majority of my accepted changes are improvements.

This seems like a reasonable plan to make progress, so LOS is a very useful number to me.

[Full confession: I am not as systematic in my approach as I just described. It's a hobby, after all. ]

[Michel]

This is an old debate which mathematicians no longer have. If you have a set X and a function P that maps certain subsets of X to the interval [0,1] and certain axioms are satisfied, we call P a probability.

The numbers that come out of Bayesian statistics could only be called probabilities in the common sense of the word if the prior were perfectly known, which is almost never the case.

The concept of "belief" is the foundation of Bayesian statistics.

https://en.wikipedia.org/wiki/Bayesian_statistics

[AlvaroBegue]

This is an old debate which mathematicians no longer have. If you have a set X and a function P that maps certain subsets of X to the interval [0,1] and certain axioms are satisfied, we call P a probability.

The numbers that come out of Bayesian statistics could only be called probabilities in the common sense of the word if the prior were perfectly known, which is almost never the case.

The concept of "belief" is the foundation of Bayesian statistics.

https://en.wikipedia.org/wiki/Bayesian_statistics

I don't know what the "common sense" of the word is. I've been a mathematician for too long, so when I hear "probability", I understand "something that satisfies Kolmogorov's axioms".

[Michel]

A probability associated with an event is something that can be measured experimentally (like the probability of throwing 6 with a dice).

"Bayesian probabilities" do not pretend to be probabilities associated with an event since they are computed from a prior which in almost all cases is not the true prior. That's is why Bayesian probabilities are to be interpreted as a "degree of belief".

You are referring to the mathematical theory of probability which of course Bayesian statistics uses. But a mathematical theory is not a scientific theory. To apply mathematics you have to codify rules how to translate mathematical concepts into real world concepts.

[AlvaroBegue]

A probability associated with an event is something that can be measured experimentally (like the probability of throwing 6 with a dice).

Ah, you are taking the point of view that by probability means "empirical probability" (a.k.a. "frequentist probability"). See a good summary of different perspectives here: https://www.ma.utexas.edu/users/mks/sta ... ility.html

"Bayesian probabilities" do not pretend to be probabilities associated with an event since they are computed from a prior which in almost all cases is not the true prior. That's is why Bayesian probabilities are to be interpreted as a "degree of belief".

Bayesian probabilities are most definitely associated with an event. I am not sure what "the true prior" means in general. Yes, their meaning is a "degree of belief", but that can be viewed as a generalization of the frequentist interpretation. If you roll a die behind a screen and you ask me what is the probability that the result was a 6, I would say 1/6. You could argue that the roll has already been cast, and it either came up a 6 or it didn't, and it doesn't make sense to talk about probabilities. But as a degree of belief, 1/6 is a perfectly good probability to work with. When new information is incorporated (like if I somehow learn the parity of the result, or you lift the screen and I can directly observe it), the probability distribution changes according to Bayes's formula.

You are referring to the mathematical theory of probability which of course Bayesian statistics uses.

No, I described the axiomatic definition of probability, which applies to both the frequentist and the Bayesian interpretations.

But a mathematical theory is not a scientific theory. To apply mathematics you have to codify rules how to translate mathematical concepts into real world concepts.

Last week a mathematician made a presentation at work where the word "reality" was used in quotes, almost as an apology for using such ill-defined concept. I prefer to call it "the R word".

[Laskos]

Even so, as non-mathematician, I find information ("probabilities") inferred from Bayesian reasoning more intuitive in making either quantitative or qualitative statements about probabilistic events than the degree of confidence in rejecting Null hypothesis. Besides that, rejecting Null hypothesis should be made with care, with the experiment pre-set to certain constraints.

Suppose a man came to you with a coin, and said "whenever heads come up I win a dollar, whenever tails come up you win a dollar". You believe the coin is fair, and start the game. Your prior for the coin is the following:



Based on that you estimate the LOS of the coin at 50.0%
After 5 tosses, the result came unfavorably, 5 heads, 0 tails.
Based on that you estimate the LOS of the coin at 55.3%

So you begin to suspect that the coin is unfair, although your suspicion is mild (55.3%). You take another, mild prior for the coin, a bit favoring heads (the man proposing the game):



Based on new prior, you re-interpret the LOS of the coin after the same 5-0 as before (no more tosses) at 98.9%. So you come to easily interpretable conclusion that the man is cheating you. Based on mild suspicion and mild prior.

That's for me is better than playing with more rigorous but vague Null hypothesis (which, in this particular case, wouldn't tell anything "rigorously and scientifically").

[Dann Corbit]

I look at Bayesian logic as a collection of facts that shed likelihood information on an event.

As new information arrives, the decisions become more and more sound.

To me, that is the real distinction. We change our decisions as new information comes forward {usually by our performing research on the subject} to select the choice that generates greatest total benefit or least possible harm, depending on what sort of outcome we are looking at.

If we have a single fact upon which to base our decision, then Bayesian logic works like any other standard method.