Does Ordo incorporate priors correctly?

[Kiudee]

I was experimenting with using prior knowledge about white advantage and draw rate in Ordo, but noticed that the posterior standard deviations were much too low. Sometimes even two orders of magnitude lower than expected.

To test this I created a very simple pgn file consisting of just 2 games. One where player A wins with white and one where player B wins with white (edit: this is just an extreme example case - but the problem happens for all possible sample sizes).
I then called Ordo as follows:

Code: Select all

ordo -N 0 -s 10000 -q -U "0,1,2,3,4,5,6,7,8,9,10" -w 50 -u 1 test.pgn

   # PLAYER    :  RATING  ERROR  POINTS  PLAYED   (%)  CFS(%)    W    D    L  D(%)
   1 A         :    2300    145     1.0       2  50.0      50    1    0    1   0.0
   2 B         :    2300    145     1.0       2  50.0     ---    1    0    1   0.0

White advantage = 50.01 +/- 0.01
Draw rate (equal opponents) = 50.00 % +/- 0.00

As the match only consists of 2 games, we would expect the posterior standard deviation of the white advantage to lie extremely close to the prior standard deviation.
But, as you can see the prior standard deviation for the white advantage is 1.0 while the posterior standard deviation is 0.01.

I went one step further and calculated the posterior standard deviation for many values in the interval [0, 80] and this is the result:


Is this a bug? If this is known already, let me know.

[Daniel Shawul]

Why are you using Ordo anyway, clearly it has inferior algorithms than bayeselo that is based on bayesian approach.
Here https://www.remi-coulom.fr/Bayesian-Elo/ Remi remid discusses some of the advantages over prior alogrithm, EloStat.
For the calculation of accurate standard deviations, there is an option to calculate the covariance matrix ( a bit slower) is not the default.
Ordo probably uses a monte carlo sampling of some sort for that, but in bayeselo you find better theory and algorithm.

Bayeselo does have the home field advantage (color) into consideration bud does not take into consideration draw ratio.
It was later extended it to take that into consideration using the Davidson model which turned out to be the best of three other
draw models.

[Kiudee]

Why are you using Ordo anyway, clearly it has inferior algorithms than bayeselo that is based on bayesian approach.
Here https://www.remi-coulom.fr/Bayesian-Elo/ Remi remid discusses some of the advantages over prior alogrithm, EloStat.
For the calculation of accurate standard deviations, there is an option to calculate the covariance matrix ( a bit slower) is not the default.
Ordo probably uses a monte carlo sampling of some sort for that, but in bayeselo you find better theory and algorithm.

Bayeselo does have the home field advantage (color) into consideration bud does not take into consideration draw ratio.
It was later extended it to take that into consideration using the Davidson model which turned out to be the best of three other
draw models.

I am very familiar with BayesElo. I am sorry if this was not clear: I am not looking for alternative software, where this feature works. I want to fix the bug in Ordo.

[Daniel Shawul]

Why are you using Ordo anyway, clearly it has inferior algorithms than bayeselo that is based on bayesian approach.
Here https://www.remi-coulom.fr/Bayesian-Elo/ Remi remid discusses some of the advantages over prior alogrithm, EloStat.
For the calculation of accurate standard deviations, there is an option to calculate the covariance matrix ( a bit slower) is not the default.
Ordo probably uses a monte carlo sampling of some sort for that, but in bayeselo you find better theory and algorithm.

Bayeselo does have the home field advantage (color) into consideration bud does not take into consideration draw ratio.
It was later extended it to take that into consideration using the Davidson model which turned out to be the best of three other
draw models.

I am very familiar with BayesElo. I am sorry if this was not clear: I am not looking for alternative software, where this feature works. I want to fix the bug in Ordo.

How are you sure it is a bug not a feature
Questions I have:
a) Does Ordo even have the concept of priors/posterieors etc? You are talking as if it has it but AFAIk it is not based on bayesian approach.
b) Does it take into consideration the color and draw ratio when calculating elos?

[Kiudee]

How are you sure it is a bug not a feature
Questions I have:
a) Does Ordo even have the concept of priors/posterieors etc? You are talking as if it has it but AFAIk it is not based on bayesian approach.
b) Does it take into consideration the color and draw ratio when calculating elos?

a) Yes, Ordo has two modes. One simulation based one and one maximum likelihood mode. If you give it a prior for drawrate or white advantage, it switches to maximum likelihood and treats these as Gaussian priors. It adds them as log likelihood terms to the total "unfitness" (that’s the term used in the code):
https://github.com/michiguel/Ordo/blob/ ... #L487-L522

b) I can recommend checking out the manual:
https://sites.google.com/site/gaviotach ... rdo-readme
It includes information about the modelling assumptions.

[Daniel Shawul]

How are you sure it is a bug not a feature
Questions I have:
a) Does Ordo even have the concept of priors/posterieors etc? You are talking as if it has it but AFAIk it is not based on bayesian approach.
b) Does it take into consideration the color and draw ratio when calculating elos?

a) Yes, Ordo has two modes. One simulation based one and one maximum likelihood mode. If you give it a prior for drawrate or white advantage, it switches to maximum likelihood and treats these as Gaussian priors. It adds them as log likelihood terms to the total "unfitness" (that’s the term used in the code):
https://github.com/michiguel/Ordo/blob/ ... #L487-L522

I am only aware it does the first i.e. it is simulation-based. For the MLE estimation, you don't have to feed anything to the elo estimator regarding
the draw rate and white advantage, that is just like asking I have the formula to calculate elos but need to be the draw rate and white advantage...bizarre.
You can certainly figure out the home advantage and draw ratio from the data, and for the latter you would need a draw model.
Bayeselo prints estimates of home advantage for you automatically, and with the modified version the draw ratio too.
I am only interested in the algorithms so forgive my mentioning of bayeselo

[Kiudee]

After reading the code very carefully, I noticed that the standard deviation shown in the result output results from passing the

Code: Select all

--simulations

flag. Ordo is then replaying the games several times to obtain error bars on all quantities.
So, in the case of white advantage and draw rate, the prior information in each simulation is so strong that all repetitions will result in very similar white advantage and draw rate values.

To sum up, what is shown in the result screen is not the posterior standard deviation, but rather a kind of residual error caused by the random games when combined with the strong prior. It is not per se a bug, but very misleading and I would say not what users would expect.