Likelihood of superiority

Discussion of chess software programming and technical issues.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
Rémi Coulom
Posts: 432
Joined: Mon Apr 24, 2006 6:06 pm
Contact:

Re: Likelihood of superiority

Post by Rémi Coulom » Wed Nov 18, 2009 11:55 am

Hart wrote:How would this compare to using a z-test on a proportion? I always thought that is how it was calculated but when I compare my numbers to yours they look very different.
When the number of games is high, the numbers should match.

For instance, with n1 = 120, n0 = 110, you have

mu = 120/230
sigma² = mu*(1-mu)/229
(mu-0.5)/sigma = 0.658568

Look it up in that table:
http://www.math.unb.ca/~knight/utility/NormTble.htm
-> something between 0.7422 and 0.7454

My program gives 0.7447, so that matches very well.

If you add a few draws, for instance 100

mu = 170/330
sigma² = (120*(1-mu)² + 100*(0.5-mu)² + 110*(0-mu)²)/(329*330)
(mu - 0.5)/sigma = 0.661198

It is almost identical.

Rémi

Michel
Posts: 2040
Joined: Sun Sep 28, 2008 11:50 pm

Re: Likelihood of superiority

Post by Michel » Wed Nov 18, 2009 4:11 pm

I always wondered why BayesElo uses Bayesian statistics.

Maximum likelihood also works for traditional frequentist statistics and then one doesn't need
to choose a prior.

This is not criticism. I am just curious to know the reason.

Rémi Coulom
Posts: 432
Joined: Mon Apr 24, 2006 6:06 pm
Contact:

Re: Likelihood of superiority

Post by Rémi Coulom » Wed Nov 18, 2009 5:05 pm

Michel wrote:I always wondered why BayesElo uses Bayesian statistics.

Maximum likelihood also works for traditional frequentist statistics and then one doesn't need
to choose a prior.

This is not criticism. I am just curious to know the reason.
Pure maximum-likelihood without a prior does not work as well as using a prior. The prior helps to regularize and avoids overfitting.

I measured quality of predictions with or without a prior on the WBEC database, and found that having a prior improved very significantly. This is particularly obvious when a player has a winning rate close to 0 or 1.

Rémi

Michel
Posts: 2040
Joined: Sun Sep 28, 2008 11:50 pm

Re: Likelihood of superiority

Post by Michel » Wed Nov 18, 2009 5:44 pm

I measured quality of predictions with or without a prior on the WBEC database, and found that having a prior improved very significantly. This is particularly obvious when a player has a winning rate close to 0 or 1.
Thank you!

I just wanted to know. :)

Post Reply