LOS (again)

Discussion of chess software programming and technical issues.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
User avatar
Ajedrecista
Posts: 1397
Joined: Wed Jul 13, 2011 7:04 pm
Location: Madrid, Spain.
Contact:

Re: LOS (again).

Post by Ajedrecista » Wed Oct 31, 2012 4:47 pm

Hi Ed:

You assign a wrong value for LOS in your code box: I calculate k = |0.5 - score|/sigma and then I calculate the area below the normal distribution density function between -infinity and k; I have to do it approximating the integral with the Simpson's composite rule because I do not know other form to do it in Fortran.

I have other programme for calculate error bars and LOS values:

Code: Select all

LOS_and_Elo_uncertainties_calculator, ® 2012.

----------------------------------------------------------------
Calculation of Elo uncertainties in a match between two engines:
----------------------------------------------------------------

(The input and output data is referred to the first engine).

Please write down non-negative integers.

Maximum number of games supported: 2147483647.

Write down the number of wins (up to 1825361100):

658

Write down the number of loses (up to 1825361100):

696

Write down the number of draws (up to 2147482293):

960

 Write down the confidence level (in percentage) between 65% and 99.9% (it will be rounded up to 0.01%):

95

Write down the clock rate of the CPU (in GHz), only for timing the elapsed time of the calculations:

3

---------------------------------------
Elo interval for 95.00 % confidence:

Elo rating difference:     -5.71 Elo

Lower rating difference:  -16.54 Elo
Upper rating difference:    5.12 Elo

Lower bound uncertainty:  -10.84 Elo
Upper bound uncertainty:   10.83 Elo
Average error:        +/-  10.83 Elo

K = (average error)*[sqrt(n)] =  521.08

Elo interval: ] -16.54,    5.12[
---------------------------------------

Number of games of the match:      2314
Score: 49.18 %
Elo rating difference:   -5.71 Elo
Draw ratio: 41.49 %

*********************************************************
Standard deviation:  1.5580 % of the points of the match.
*********************************************************

 Error bars were calculated with two-sided tests; values are rounded up to 0.01 Elo, or 0.01 in the case of K.

-------------------------------------------------------------------
Calculation of likelihood of superiority (LOS) in a one-sided test:
-------------------------------------------------------------------

LOS (taking into account draws) is always calculated, if possible.

LOS &#40;not taking into account draws&#41; is only calculated if wins + loses < 16001.

LOS &#40;average value&#41; is calculated only when LOS &#40;not taking into account draws&#41; is calculated.
______________________________________________

LOS&#58;  15.08 % &#40;taking into account draws&#41;.
LOS&#58;  15.10 % &#40;not taking into account draws&#41;.
LOS&#58;  15.09 % &#40;average value&#41;.
______________________________________________

These values of LOS are rounded up to 0.01%

End of the calculations. Approximated elapsed time&#58;   74 ms.

Thanks for using LOS_and_Elo_uncertainties_calculator. Press Enter to exit.
I get a LOS value of ~ 15.08% using my own method; the other LOS value ~ 15.1% is done in other way proposed by Rémi Coulom in the last equation of this post. As you can see, 15.08% and 15.1% are enough close between them to almost consider both values as the same.

------------------------

I will try to compute LOS tables from 2000 games to 100000 games in steps of 1000 games (I mean: 2000, 3000, 4000, ..., 99000 and 100000 games) fixing the draw ratio to 32%, just as CPW tables. Please do not forget that they are only (good) approximations of the trinomial distribution.

The generation of these 99 tables will take a while (maybe a few hours), so please be patient. I have the intention of uploading the notepads here for everyone.

Regarding your last post, I do not have experience in engine testing, so all my efforts to help you will be useless.
Rebel wrote:Is it possible for you to email those larger tables? I like to put them on my website as a reference.
Thank you very much for your confidence! :)

Regards from Spain.

Ajedrecista.

User avatar
Ajedrecista
Posts: 1397
Joined: Wed Jul 13, 2011 7:04 pm
Location: Madrid, Spain.
Contact:

99 LOS tables.

Post by Ajedrecista » Wed Oct 31, 2012 6:42 pm

Hi again:
Ajedrecista wrote:I will try to compute LOS tables from 2000 games to 100000 games in steps of 1000 games (I mean: 2000, 3000, 4000, ..., 99000 and 100000 games) fixing the draw ratio to 32%, just as CPW tables. Please do not forget that they are only (good) approximations of the trinomial distribution.

The generation of these 99 tables will take a while (maybe a few hours), so please be patient. I have the intention of uploading the notepads here for everyone.
I have computed 99 LOS tables from 2000 games up to 100000 games, in steps of 1000 games, as promised. I fixed the draw ratio to 32% (the same as the tables that are hosted at CPW) although very similar LOS values should be obtained with different draw ratios that are not extreme (I mean: not draw ratios of 99% or 100%, for example).

LOS_tables (0.24 MB)

I hope no typos in the notepads; I include a Readme.txt file with little info that must be considered.

Regards from Spain.

Ajedrecista.

User avatar
Rebel
Posts: 4663
Joined: Thu Aug 18, 2011 10:04 am

Re: 99 LOS tables.

Post by Rebel » Wed Oct 31, 2012 10:40 pm

Thank you, I will put the most important ones on a few web-pages.

For myself I am using Rémi Coulom's formula now.

User avatar
Laskos
Posts: 9417
Joined: Wed Jul 26, 2006 8:21 pm
Full name: Kai Laskos

Re: 99 LOS tables.

Post by Laskos » Thu Nov 01, 2012 12:38 am

Rebel wrote:Thank you, I will put the most important ones on a few web-pages.

For myself I am using Rémi Coulom's formula now.
What is Remi's formula? For LOS in a single match I used, if I am not wrong

LOS = (Sum_over_i [Binomial[wins + losses, i], {i from 0 to wins - 1}] + 0.5*Binomial[wins + losses, wins]) / 2^(wins + losses)

I tried without success to sum it up to a Hypergeometric 2F1 and such.

Kai

User avatar
Rebel
Posts: 4663
Joined: Thu Aug 18, 2011 10:04 am

Re: 99 LOS tables.

Post by Rebel » Thu Nov 01, 2012 7:31 am


User avatar
Rebel
Posts: 4663
Joined: Thu Aug 18, 2011 10:04 am

Re: 99 LOS tables.

Post by Rebel » Thu Nov 01, 2012 11:04 am

LOS tables from 2000 games and on are up.

http://www.top-5000.nl/los.htm

User avatar
Ajedrecista
Posts: 1397
Joined: Wed Jul 13, 2011 7:04 pm
Location: Madrid, Spain.
Contact:

Re: 99 LOS tables.

Post by Ajedrecista » Thu Nov 01, 2012 11:39 am

Hi Ed:
Rebel wrote:LOS tables from 2000 games and on are up.

http://www.top-5000.nl/los.htm
Thanks for the upload and the credits!

I must say that columns for 10000 and 25000 games are the same in your web: you wrongly copied the file with 10000 games twice and the true column for 25000 games is not present right now. I hope a quick fix.

I forgot to write in Readme.txt file that the draw ratio was fixed to 32% but fortunately I included that info in each notepad with LOS tables. Once again, thank you very much for your confidence! :)

Regards from Spain.

Ajedrecista.

User avatar
Rebel
Posts: 4663
Joined: Thu Aug 18, 2011 10:04 am

Re: 99 LOS tables.

Post by Rebel » Thu Nov 01, 2012 2:20 pm

Corrected the error, tx for pointing out.

Speaking of errors, Bayesian elo and Elostat display an error-margin. Is the formula known and undisputed? The page deserves such a reference table.

User avatar
Ajedrecista
Posts: 1397
Joined: Wed Jul 13, 2011 7:04 pm
Location: Madrid, Spain.
Contact:

Re: about error bars.

Post by Ajedrecista » Thu Nov 01, 2012 4:22 pm

Hi again:
Rebel wrote:Speaking of errors, Bayesian elo and Elostat display an error-margin. Is the formula known and undisputed? The page deserves such a reference table.
I am not sure at all, but maybe BayesElo and EloSTAT differ a little; I use a method that gives similar error bars than EloSTAT if I am not wrong. Again I use the same mean and standard deviation of the same normal distribution that I used for generating LOS tables.

Of course error bars depend on the confidence level you want (95%, 99%, etc.; a higher confidence implies a larger error bar for the same match) but I do not feel confident of calculate error bars in this case because the formula may be disputed (I mean: there are probably several approaches to calculate error bars). You should ask to other people to obtain more valuable help.

Regards from Spain.

Ajedrecista.

John_F
Posts: 44
Joined: Wed Jan 27, 2010 1:58 pm

Re: 99 LOS tables.

Post by John_F » Thu Nov 29, 2012 4:40 pm

Does anyone know of a spreadsheet formula to calculate LOS? (Like for example, to determine the LOS of the winner in a match between two programs)

Post Reply