Calculating the LOS (likelihood of superiority) from results

[mohzus]

Hi guys,
I found a formula to calculate the LOS at http://chessprogramming.wikispaces.com/Match+Statistics, basically it's LOS=(1/2)*(1+erf (x/sqrt 2)) where x is defined as score_difference/(N/(1-draw_ratio)) where N is the total number of games. This formula is supposed to be a good approximation when the number of games is large.
So I've tested a bit this formula with very few games (not more than 300) and then I tried to do it once for a large number of games, around 40 k games.

The data is the following (taken out directly from the fishtest, the LOS is supposed to be close to 17% according to the fishtest calculations):

Code: Select all

Total: 38946 W: 5975 L: 6077 D: 26894

.
From this I calculated

Code: Select all

Draw rate: 0,69
Score difference: -102
x= -102/(38946*(1- 0,69)) =  -0.0084633
LOS=(1/2)*(1+erf (-0.0084633 /sqrt 2 )) =0.4966

. Note that in order to perform the calculations, I've kept more digits than the ones I've written here.
Conclusion: the LOS that I've calculated is totally off from 0.17...
What am I doing wrong?

[Laskos]

Hi guys,
I found a formula to calculate the LOS at http://chessprogramming.wikispaces.com/Match+Statistics, basically it's LOS=(1/2)*(1+erf (x/sqrt 2)) where x is defined as score_difference/(N/(1-draw_ratio)) where N is the total number of games. This formula is supposed to be a good approximation when the number of games is large.
So I've tested a bit this formula with very few games (not more than 300) and then I tried to do it once for a large number of games, around 40 k games.

The data is the following (taken out directly from the fishtest, the LOS is supposed to be close to 17% according to the fishtest calculations):

Code: Select all

Total: 38946 W: 5975 L: 6077 D: 26894

.
From this I calculated

Code: Select all

Draw rate: 0,69
Score difference: -102
x= -102/(38946*(1- 0,69)) =  -0.0084633
LOS=(1/2)*(1+erf (-0.0084633 /sqrt 2 )) =0.4966

. Note that in order to perform the calculations, I've kept more digits than the ones I've written here.
Conclusion: the LOS that I've calculated is totally off from 0.17...
What am I doing wrong?

They forgot a sqrt there in Wiki, and added useless draw ratio.
"score_difference/(N/(1-draw_ratio))" should be score_difference/sqrt(N*(1-draw_ratio)). Number of draws is irrelevant:

LOS = (1 + erf[(wins - losses)/(2*(wins+losses))^0.5])/2
By the way, it's my formula.
In your particular case, the exact LOS is 0.176424329128482367, that Erf approximation is 0.176414.

[AlvaroBegue]

I wrote a tiny C++11 program to compute ELO difference and LOS from W/L/D counts:

Code: Select all

#include <cstdio>
#include <cstdlib>
#include <cmath>

int main&#40;int argc, char **argv&#41; &#123;
  if &#40;argc != 4&#41; &#123;
    std&#58;&#58;printf&#40;"Wrong number of arguments.\n\nUsage&#58;%s <wins> <losses> <draws>\n", argv&#91;0&#93;);
    return 1;
  &#125;
  int wins = std&#58;&#58;atoi&#40;argv&#91;1&#93;);
  int losses = std&#58;&#58;atoi&#40;argv&#91;2&#93;);
  int draws = std&#58;&#58;atoi&#40;argv&#91;3&#93;);
  
  double games = wins + losses + draws;
  std&#58;&#58;printf&#40;"Number of games&#58; %g\n", games&#41;;
  double winning_fraction = &#40;wins + 0.5*draws&#41; / games;
  std&#58;&#58;printf&#40;"Winning fraction&#58; %g\n", winning_fraction&#41;;
  double elo_difference = -std&#58;&#58;log&#40;1.0/winning_fraction-1.0&#41;*400.0/std&#58;&#58;log&#40;10.0&#41;;
  std&#58;&#58;printf&#40;"Elo difference&#58; %+g\n", elo_difference&#41;;
  double los = .5 + .5 * std&#58;&#58;erf&#40;&#40;wins-losses&#41;/std&#58;&#58;sqrt&#40;2.0*&#40;wins+losses&#41;));
  std&#58;&#58;printf&#40;"LOS&#58; %g\n", los&#41;;
&#125;

Feel free to use it.

[mohzus]

Thank you very much guys!

[Gerd Isenberg]

Hi guys,
I found a formula to calculate the LOS at http://chessprogramming.wikispaces.com/Match+Statistics, basically it's LOS=(1/2)*(1+erf (x/sqrt 2)) where x is defined as score_difference/(N/(1-draw_ratio)) where N is the total number of games. This formula is supposed to be a good approximation when the number of games is large.
So I've tested a bit this formula with very few games (not more than 300) and then I tried to do it once for a large number of games, around 40 k games.

The data is the following (taken out directly from the fishtest, the LOS is supposed to be close to 17% according to the fishtest calculations):

Code: Select all

Total: 38946 W: 5975 L: 6077 D: 26894

.
From this I calculated

Code: Select all

Draw rate: 0,69
Score difference: -102
x= -102/(38946*(1- 0,69)) =  -0.0084633
LOS=(1/2)*(1+erf (-0.0084633 /sqrt 2 )) =0.4966

. Note that in order to perform the calculations, I've kept more digits than the ones I've written here.
Conclusion: the LOS that I've calculated is totally off from 0.17...
What am I doing wrong?

They forgot a sqrt there in Wiki, and added useless draw ratio.
"score_difference/(N/(1-draw_ratio))" should be score_difference/sqrt(N*(1-draw_ratio)). Number of draws is irrelevant:

LOS = (1 + erf[(wins - losses)/(2*(wins+losses))^0.5])/2
By the way, it's my formula.
In your particular case, the exact LOS is 0.176424329128482367, that Erf approximation is 0.176414.

Thanks for the correction. Can you review it again whether it is now correct? I am illiterate in statistics, and Edmund seems actually inactive as cpw editor.

[Rémi Coulom]

Thanks for the correction. Can you review it again whether it is now correct? I am illiterate in statistics, and Edmund seems actually inactive as cpw editor.

I don't have time to edit the wiki, but I'd like to make some remarks.

This calculation becomes very inefficient for larger number of games. In this case the Normal Distribution can give a good approximation.

Well, the calculation can be done cleverly, and the Normal approximation is not really required.

http://www.talkchess.com/forum/viewtopi ... 82&t=30624
http://www.talkchess.com/forum/viewtopi ... 05&t=30624
http://www.talkchess.com/forum/viewtopi ... 30&t=30624

It is important to note that LOS does not depend on the number of draws.

These calculations don't make a difference whether the game results were obtained when playing Black or White. It is a good approximation when the two players played the same number of games with each color.

Rémi

[Laskos]

Hi guys,
I found a formula to calculate the LOS at http://chessprogramming.wikispaces.com/Match+Statistics, basically it's LOS=(1/2)*(1+erf (x/sqrt 2)) where x is defined as score_difference/(N/(1-draw_ratio)) where N is the total number of games. This formula is supposed to be a good approximation when the number of games is large.
So I've tested a bit this formula with very few games (not more than 300) and then I tried to do it once for a large number of games, around 40 k games.

The data is the following (taken out directly from the fishtest, the LOS is supposed to be close to 17% according to the fishtest calculations):

Code: Select all

Total: 38946 W: 5975 L: 6077 D: 26894

.
From this I calculated

Code: Select all

Draw rate: 0,69
Score difference: -102
x= -102/(38946*(1- 0,69)) =  -0.0084633
LOS=(1/2)*(1+erf (-0.0084633 /sqrt 2 )) =0.4966

. Note that in order to perform the calculations, I've kept more digits than the ones I've written here.
Conclusion: the LOS that I've calculated is totally off from 0.17...
What am I doing wrong?

They forgot a sqrt there in Wiki, and added useless draw ratio.
"score_difference/(N/(1-draw_ratio))" should be score_difference/sqrt(N*(1-draw_ratio)). Number of draws is irrelevant:

LOS = (1 + erf[(wins - losses)/(2*(wins+losses))^0.5])/2
By the way, it's my formula.
In your particular case, the exact LOS is 0.176424329128482367, that Erf approximation is 0.176414.

Thanks for the correction. Can you review it again whether it is now correct? I am illiterate in statistics, and Edmund seems actually inactive as cpw editor.

Yes, now it seems correct, although I would explicitly show that the number of draws doesn't matter.

[Gerd Isenberg]

Hi guys,
I found a formula to calculate the LOS at http://chessprogramming.wikispaces.com/Match+Statistics, basically it's LOS=(1/2)*(1+erf (x/sqrt 2)) where x is defined as score_difference/(N/(1-draw_ratio)) where N is the total number of games. This formula is supposed to be a good approximation when the number of games is large.
So I've tested a bit this formula with very few games (not more than 300) and then I tried to do it once for a large number of games, around 40 k games.

The data is the following (taken out directly from the fishtest, the LOS is supposed to be close to 17% according to the fishtest calculations):

Code: Select all

Total: 38946 W: 5975 L: 6077 D: 26894

.
From this I calculated

Code: Select all

Draw rate: 0,69
Score difference: -102
x= -102/(38946*(1- 0,69)) =  -0.0084633
LOS=(1/2)*(1+erf (-0.0084633 /sqrt 2 )) =0.4966

. Note that in order to perform the calculations, I've kept more digits than the ones I've written here.
Conclusion: the LOS that I've calculated is totally off from 0.17...
What am I doing wrong?

They forgot a sqrt there in Wiki, and added useless draw ratio.
"score_difference/(N/(1-draw_ratio))" should be score_difference/sqrt(N*(1-draw_ratio)). Number of draws is irrelevant:

LOS = (1 + erf[(wins - losses)/(2*(wins+losses))^0.5])/2
By the way, it's my formula.
In your particular case, the exact LOS is 0.176424329128482367, that Erf approximation is 0.176414.

Thanks for the correction. Can you review it again whether it is now correct? I am illiterate in statistics, and Edmund seems actually inactive as cpw editor.

Yes, now it seems correct, although I would explicitly show that the number of draws doesn't matter.

I will try to do that during the next days.

[Isaac]

I would like to mention that the formula given in the wiki page (http://chessprogramming.wikispaces.com/Match+Statistics , which can be written as los=(1+erf ((wins-losses)/ (2*(wins+losses))**0.5) )/2. into a Fortran program) is not equivalent to the LOS given by Rémi Coulom in the first link he gives in this thread, namely http://www.talkchess.com/forum/viewtopi ... 82&t=30624.

For example if you tell the program that there is 1 win and 0 loss, the Rémi's program gives the value 0.75 while the formula on wiki programming gives 0.84.

[AlvaroBegue]

I would like to mention that the formula given in the wiki page (http://chessprogramming.wikispaces.com/Match+Statistics , which can be written as los=(1+erf ((wins-losses)/ (2*(wins+losses))**0.5) )/2. into a Fortran program) is not equivalent to the LOS given by Rémi Coulom in the first link he gives in this thread, namely http://www.talkchess.com/forum/viewtopi ... 82&t=30624.

For example if you tell the program that there is 1 win and 0 loss, the Rémi's program gives the value 0.75 while the formula on wiki programming gives 0.84.

The erf formula is an approximation that works well with large numbers of games. Using it with 1 win and 0 losses is not a good idea. But we are trying to answer questions about whether 10,000 games or 30,000 games is enough to get statistical significance, and that's done perfectly well by the erf formula.