about error margins?

Discussion of chess software programming and technical issues.

Moderators: hgm, Harvey Williamson, bob

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
User avatar
Kempelen
Posts: 620
Joined: Fri Feb 08, 2008 9:44 am
Location: Madrid - Spain
Contact:

about error margins?

Post by Kempelen » Wed Aug 01, 2012 3:20 pm

Could someone explain to me where the current formulas to calculate error margins according to number of games cames from? (i.e. +-10 elo points for 4000 games). I mean, Why those numbers and no others?. What observations has been done to arrive to that conclusions?
thx Fermin
Fermin Serrano
Author of 'Rodin' engine
http://sites.google.com/site/clonfsp/

Daniel Shawul
Posts: 3724
Joined: Tue Mar 14, 2006 10:34 am
Location: Ethiopia
Contact:

Re: about error margins?

Post by Daniel Shawul » Wed Aug 01, 2012 4:59 pm

Kempelen wrote:Could someone explain to me where the current formulas to calculate error margins according to number of games cames from? (i.e. +-10 elo points for 4000 games). I mean, Why those numbers and no others?. What observations has been done to arrive to that conclusions?
thx Fermin
I think you would need an estimate of the sample standard deviation to give specific values. Otherwise all you can say is that for every doubling of number of games ,the error margin would be halved... For 95% confidence i.e 2 sigma you have , mean +/- 2 * sd / sqrt(n). Then from winning percentage and elo relation you can tell how much the error margin is interms of elo. If you have a record of games, like WLWWWDDLLLWL etc , you can estimate the sample mean and sd from it.

User avatar
hgm
Posts: 23510
Joined: Fri Mar 10, 2006 9:06 am
Location: Amsterdam
Full name: H G Muller
Contact:

Re: about error margins?

Post by hgm » Wed Aug 01, 2012 5:26 pm

Kempelen wrote:What observations has been done to arrive to that conclusions?
No observations have to be done, because this is simple mathematics. One also doesn't do observations to establish that 5+5=10...

It follows from the definition of standard deviation, confidence interval and the fact that the games are independent.

Sven
Posts: 3820
Joined: Thu May 15, 2008 7:57 pm
Location: Berlin, Germany
Full name: Sven Schüle
Contact:

Re: about error margins?

Post by Sven » Wed Aug 01, 2012 9:41 pm

Daniel Shawul wrote:I think you would need an estimate of the sample standard deviation to give specific values. Otherwise all you can say is that for every doubling of number of games ,the error margin would be halved...
In principle I agree, but the sample standard deviation is already an estimate (of the "real" standard deviation which is unknown) so you don't need to estimate it again. And for every doubling of the number of games the error margin is divided by sqrt(2), or for every multiplying of the number of games by four the error margin is halved.

Sven

Daniel Shawul
Posts: 3724
Joined: Tue Mar 14, 2006 10:34 am
Location: Ethiopia
Contact:

Re: about error margins?

Post by Daniel Shawul » Wed Aug 01, 2012 10:35 pm

Sven Schüle wrote:
Daniel Shawul wrote:I think you would need an estimate of the sample standard deviation to give specific values. Otherwise all you can say is that for every doubling of number of games ,the error margin would be halved...
In principle I agree, but the sample standard deviation is already an estimate (of the "real" standard deviation which is unknown) so you don't need to estimate it again. And for every doubling of the number of games the error margin is divided by sqrt(2), or for every multiplying of the number of games by four the error margin is halved.

Sven
Just left that mistake for mr nitpicker to find it :) Correction people,it is when number of games is _quadrupled_ not doubled you get s.e halved !

But I was not saying he should use the population sd but just the sample sd. Note that he mentioned only number of games and associated error of margin. It is impossilbe to determine error of margin from number of games alone so he needs to estimate the standard deviation from the sample of games. 4000 games is pretty high so sample or population statistic don't matter much even if the later is known. It is a good question though, as I have noticed some say (including myself) that 4000 games is a +/10 elo or so which is not quite right. Probably that is what Fermin meant by observations. Most of the time you would get a +/- 10 elo from bayeselo at around 3-4 thousand games.. For example a 50% winning percentage could be from a WLWLWLWLWL or DDDDDDDDDD. The latter has a big sd ,but the latter has 0 sd!

Code: Select all

sd = sqrt(wins * (1 - m)^2 + losses * (0 - m)^2 + draws * (0.5 - m)^2) / sqrt(n - 1)
If the draw ratio is 0, it is possible to tell the error margin from the number of games and winning percentage alone.

User avatar
Laskos
Posts: 9324
Joined: Wed Jul 26, 2006 8:21 pm
Full name: Kai Laskos

Re: about error margins?

Post by Laskos » Wed Aug 01, 2012 10:49 pm

Daniel Shawul wrote:
Sven Schüle wrote:
Daniel Shawul wrote:I think you would need an estimate of the sample standard deviation to give specific values. Otherwise all you can say is that for every doubling of number of games ,the error margin would be halved...
In principle I agree, but the sample standard deviation is already an estimate (of the "real" standard deviation which is unknown) so you don't need to estimate it again. And for every doubling of the number of games the error margin is divided by sqrt(2), or for every multiplying of the number of games by four the error margin is halved.

Sven
Just left that mistake for mr nitpicker to find it :) Correction people,it is when number of games is _quadrupled_ not doubled you get s.e halved !

But I was not saying he should use the population sd but just the sample sd. Note that he mentioned only number of games and associated error of margin. It is impossilbe to determine error of margin from number of games alone so he needs to estimate the standard deviation from the sample of games. 4000 games is pretty high so sample or population statistic don't matter much even if the later is known. It is a good question though, as I have noticed some say (including myself) that 4000 games is a +/10 elo or so which is not quite right. Probably that is what Fermin meant by observations. Most of the time you would get a +/- 10 elo from bayeselo at around 3-4 thousand games.. For example a 50% winning percentage could be from a WLWLWLWLWL or DDDDDDDDDD. The latter has a big sd ,but the latter has 0 sd!

Code: Select all

sd = sqrt(wins * (1 - m)^2 + losses * (0 - m)^2 + draws * (0.5 - m)^2) / sqrt(n - 1)
If the draw ratio is 0, it is possible to tell the error margin from the number of games and winning percentage alone.
DDDD... gives 0 sd only in the approximation you use.

Kai

Daniel Shawul
Posts: 3724
Joined: Tue Mar 14, 2006 10:34 am
Location: Ethiopia
Contact:

Re: about error margins?

Post by Daniel Shawul » Wed Aug 01, 2012 11:03 pm

How? Note that I am calculating sd of winning percentage. When you go to elo calculation with bayeselo, there is ofcourse elodraw and eloadvantage. So a DDDDDD is more variable when you consider those things... Anyway I was just trying to demostrate why one can't tell margin of error of elo from the number of games and winning percentage alone.

User avatar
Laskos
Posts: 9324
Joined: Wed Jul 26, 2006 8:21 pm
Full name: Kai Laskos

Re: about error margins?

Post by Laskos » Wed Aug 01, 2012 11:25 pm

Daniel Shawul wrote:How? Note that I am calculating sd of winning percentage. When you go to elo calculation with bayeselo, there is ofcourse elodraw and eloadvantage. So a DDDDDD is more variable when you consider those things... Anyway I was just trying to demostrate why one can't tell margin of error of elo from the number of games and winning percentage alone.
That thing you wrote is the error (SD) in the normal approximation of the trinomial. You can still use it for 10 draws match, but keep in mind that after that match, probability of W,L = 1/13, for D = 11/13, and input these into your formula.

Kai

Daniel Shawul
Posts: 3724
Joined: Tue Mar 14, 2006 10:34 am
Location: Ethiopia
Contact:

Re: about error margins?

Post by Daniel Shawul » Wed Aug 01, 2012 11:46 pm

Laskos wrote:
Daniel Shawul wrote:How? Note that I am calculating sd of winning percentage. When you go to elo calculation with bayeselo, there is ofcourse elodraw and eloadvantage. So a DDDDDD is more variable when you consider those things... Anyway I was just trying to demostrate why one can't tell margin of error of elo from the number of games and winning percentage alone.
That thing you wrote is the error (SD) in the normal approximation of the trinomial. You can still use it for 10 draws match, but keep in mind that after that match, probability of W,L = 1/13, for D = 11/13, and input these into your formula.

Kai
The probability of a W,L would surely be lowered a lot after that odd observation.. That formula is for calculating standard deviation of a given sample that does not assume probablities for WDL. The rewards are fixed at 0,0.5,1 ofcourse. This was just a quick example , but I know it does not directly translate to elo because there you have mix of players with different strength, white elo advantage etc..

User avatar
Laskos
Posts: 9324
Joined: Wed Jul 26, 2006 8:21 pm
Full name: Kai Laskos

Re: about error margins?

Post by Laskos » Thu Aug 02, 2012 12:12 am

Daniel Shawul wrote:
Laskos wrote:
Daniel Shawul wrote:How? Note that I am calculating sd of winning percentage. When you go to elo calculation with bayeselo, there is ofcourse elodraw and eloadvantage. So a DDDDDD is more variable when you consider those things... Anyway I was just trying to demostrate why one can't tell margin of error of elo from the number of games and winning percentage alone.
That thing you wrote is the error (SD) in the normal approximation of the trinomial. You can still use it for 10 draws match, but keep in mind that after that match, probability of W,L = 1/13, for D = 11/13, and input these into your formula.

Kai
The probability of a W,L would surely be lowered a lot after that odd observation.. That formula is for calculating standard deviation of a given sample that does not assume probablities for WDL. The rewards are fixed at 0,0.5,1 ofcourse. This was just a quick example , but I know it does not directly translate to elo because there you have mix of players with different strength, white elo advantage etc..
I think the formula you wrote was derived using normal approximation and probabilities, which were translated into games by multiplying times n. The formula gives the SD as percentages, then you can convert to Elo by using the logistic. That 1/13 was derived from (w+1)/(n+3), for w=0, n=10 (10 draws, n=10, d=10)

Kai

Post Reply