best way to determine elos of a group

[Michel]

@Daniel,

How do you calculate the error bars?

I seem to recall BayesElo does it by default by assuming that the elos of the other engines are fixed. But then of course the error bar does not represent the variance of the absolute elo.

The correct way to do it is to invert the Hessian of the log likelihood (you have to delete the row and column corresponding to the anchor engine).

Ordo does it by simulation, which is also correct.

[Daniel Shawul]

@Daniel,

How do you calculate the error bars?

I seem to recall BayesElo does it by default by assuming that the elos of the other engines are fixed. But then of course the error bar does not represent the variance of the absolute elo.

The correct way to do it is to invert the Hessian of the log likelihood (you have to delete the row and column corresponding to the anchor engine).

That is what do. Here it is in code

Code: Select all

 816 //variance
 817 static void calc_variance() {
 818     double sd;
 819     covariance.assign(RSIZE * RSIZE,0.0);
 820     for(unsigned i = 0;i < n_players;i++) {
 821         sd = covariance[pair(i,i)] = (-1 / hessian[pair(i,i)]);
 822         sd = sqrt(sd);
 823         players[i].deltau = 1.96 * sd;
 824         players[i].deltal = 1.96 * sd;
 825     }
 826 }

I thought that is what bayeselo does too. I will run the ratings with bayeselo but last time i checked my rating tool gives same results as bayeselo when draws are ignored.

Ordo does it by simulation, which is also correct.

[Michel]

@Daniel,

How do you calculate the error bars?

I seem to recall BayesElo does it by default by assuming that the elos of the other engines are fixed. But then of course the error bar does not represent the variance of the absolute elo.

The correct way to do it is to invert the Hessian of the log likelihood (you have to delete the row and column corresponding to the anchor engine).

That is what do. Here it is in code

Code: Select all

 816 //variance
 817 static void calc_variance() {
 818     double sd;
 819     covariance.assign(RSIZE * RSIZE,0.0);
 820     for(unsigned i = 0;i < n_players;i++) {
 821         sd = covariance[pair(i,i)] = (-1 / hessian[pair(i,i)]);
 822         sd = sqrt(sd);
 823         players[i].deltau = 1.96 * sd;
 824         players[i].deltal = 1.96 * sd;
 825     }
 826 }

I thought that is what bayeselo does too. I will run the ratings with bayeselo but last time i checked my rating tool gives same results as bayeselo when draws are ignored.

Ordo does it by simulation, which is also correct.

Do I read it correctly that you only invert the diagonal of the Hessian matrix? In principle this is wrong (it amounts to assuming that the other elo's are fixed). You have to invert the full matrix (minus the row/column corresponding to the anchor engine), and then take the diagonal elements.

[Daniel Shawul]

Ah...I see, I do use the same assumption as bayeselo for calculating error bars. However I do the full hessian matrix inversion
when calculating likelihood of superiority (LOS). I am a bit rusty with the code since it has been too long.

Code: Select all

 827 //covariance = inverse of information matrix (negative hessian)
 828 static void calc_covariance() {
 829     unsigned i;
 830     for(i = 0;i < RSIZE * RSIZE;i++)
 831         covariance[i] = -hessian[i];
 832 
 833     //lu decompose with partial pivoting
 834     vector<int> pivot(n_players);
 835     Doolittle_LU(&covariance[0],&pivot[0],n_players);
 836 
 837     //construct A matrix
 838     vector<double> A(RSIZE * RSIZE,-1.0 / n_players);
 839     for(i = 0;i < n_players;i++)
 840         A[pair(i,i)] += 1;
 841 
 842     //solve for AC
 843     vector<double> AC(RSIZE * RSIZE,0.0);
 844     for(i = 0;i < n_players;i++) {
 845         int Index = i * n_players;
 846         Doolittle_Solve(&covariance[0], &A[Index], &pivot[0],&AC[Index],n_players);
 847     }
 848     //multiply AC by A to get covariance matrix
 849     Mul_Transpose(&AC[0],&A[0],&covariance[0],n_players);
 850 
 851     for(unsigned i = 0;i < n_players;i++) {
 852         for(unsigned j = 0;j < n_players;j++) {
 853             print("%d %d. %3d",covariance[pair(i,j)]);
 854         }
 855         print("\n");
 856     }
 857 }
 

[Michel]

Ah...I see, I do use the same assumption as bayeselo for calculating error bars. However I do the full hessian matrix inversion
when calculating likelihood of superiority (LOS). I am a bit rusty with the code since it has been too long.

Code: Select all

 827 //covariance = inverse of information matrix (negative hessian)
 828 static void calc_covariance() {
 829     unsigned i;
 830     for(i = 0;i < RSIZE * RSIZE;i++)
 831         covariance[i] = -hessian[i];
 832 
 833     //lu decompose with partial pivoting
 834     vector<int> pivot(n_players);
 835     Doolittle_LU(&covariance[0],&pivot[0],n_players);
 836 
 837     //construct A matrix
 838     vector<double> A(RSIZE * RSIZE,-1.0 / n_players);
 839     for(i = 0;i < n_players;i++)
 840         A[pair(i,i)] += 1;
 841 
 842     //solve for AC
 843     vector<double> AC(RSIZE * RSIZE,0.0);
 844     for(i = 0;i < n_players;i++) {
 845         int Index = i * n_players;
 846         Doolittle_Solve(&covariance[0], &A[Index], &pivot[0],&AC[Index],n_players);
 847     }
 848     //multiply AC by A to get covariance matrix
 849     Mul_Transpose(&AC[0],&A[0],&covariance[0],n_players);
 850 
 851     for(unsigned i = 0;i < n_players;i++) {
 852         for(unsigned j = 0;j < n_players;j++) {
 853             print("%d %d. %3d",covariance[pair(i,j)]);
 854         }
 855         print("\n");
 856     }
 857 }
 

This appears to be more or less correct. I assume that somewhere you delete a row and column from the Hessian matrix (the full Hessian matrix is singular)?

[dragontamer5788]

Here is the full ratings table with error bars:

http://scorpiozero.ddns.net/scorpiozero ... atings.txt

Hmm, I looked over the data and items aren't obvious to me.

1 ID-146 557 46 46 200 55.8% 504 35.5%

My guess is:

Rank -- ID -- Elo -- ?? -- ?? -- Number of Matches -- Win Percent? -- Elo-lower bound? -- ??

I'm probably very wrong with my above guesses.

[Ajedrecista]

Hello:

Here is the full ratings table with error bars:

http://scorpiozero.ddns.net/scorpiozero ... atings.txt

Hmm, I looked over the data and items aren't obvious to me.

1 ID-146 557 46 46 200 55.8% 504 35.5%

My guess is:

Rank -- ID -- Elo -- ?? -- ?? -- Number of Matches -- Win Percent? -- Elo-lower bound? -- ??

I'm probably very wrong with my above guesses.

I would say:

Code: Select all

Rank -- ID -- Elo -- Upper error margin -- Lower error margin -- Score (%) -- Average opponents' Elo -- Draws (%)

Score = (wins + 0.5*draws)/games = (wins + 0.5*draws)/(wins + draws + loses).

Error margins in Elo (Bayeselo), probably up to 1.96-sigma (95% confidence) or 2-sigma (~ 95.45% confidence) if a normal distribution is used somewhere, which I do not know.

Daniel will know the answer, of course.

Regards from Spain.

Ajedrecista.

[Daniel Shawul]

Ah...I see, I do use the same assumption as bayeselo for calculating error bars. However I do the full hessian matrix inversion
when calculating likelihood of superiority (LOS). I am a bit rusty with the code since it has been too long.

Code: Select all

 827 //covariance = inverse of information matrix (negative hessian)
 828 static void calc_covariance() {
 829     unsigned i;
 830     for(i = 0;i < RSIZE * RSIZE;i++)
 831         covariance[i] = -hessian[i];
 832 
 833     //lu decompose with partial pivoting
 834     vector<int> pivot(n_players);
 835     Doolittle_LU(&covariance[0],&pivot[0],n_players);
 836 
 837     //construct A matrix
 838     vector<double> A(RSIZE * RSIZE,-1.0 / n_players);
 839     for(i = 0;i < n_players;i++)
 840         A[pair(i,i)] += 1;
 841 
 842     //solve for AC
 843     vector<double> AC(RSIZE * RSIZE,0.0);
 844     for(i = 0;i < n_players;i++) {
 845         int Index = i * n_players;
 846         Doolittle_Solve(&covariance[0], &A[Index], &pivot[0],&AC[Index],n_players);
 847     }
 848     //multiply AC by A to get covariance matrix
 849     Mul_Transpose(&AC[0],&A[0],&covariance[0],n_players);
 850 
 851     for(unsigned i = 0;i < n_players;i++) {
 852         for(unsigned j = 0;j < n_players;j++) {
 853             print("%d %d. %3d",covariance[pair(i,j)]);
 854         }
 855         print("\n");
 856     }
 857 }
 

This appears to be more or less correct. I assume that somewhere you delete a row and column from the Hessian matrix (the full Hessian matrix is singular)?

Yes, I think i take care of that during partial pivoting iirc.
I checked bayeselo code now and it seems there is an option to calculate the deltas from either the variance or covariance matrix.
The default is to use the diagonal of the covariance matrix which is why you thought it does not have that option.

Code: Select all

variance ........ compute intervals with the diagonal of the Hessian
covariance ...... compute intervals with the full Hessian

Here are ratings calculated with bayeselo. The "scales" are different from what i use so the elo numbers differ but the order is almost the same.

Default delta calculated with inverse of diagonal of hessian

Code: Select all

Rank Name          Elo    +    - games score oppo. draws 
   1 scorpio-146   415   40   40   200   56%   379   36% 
   2 scorpio-147   383   56   56   100   45%   415   34% 
   3 scorpio-145   374   40   40   200   50%   375   38% 
   4 scorpio-134   347   39   39   200   55%   317   43% 
   5 scorpio-144   335   39   39   200   48%   350   40% 
   6 scorpio-141   329   39   39   200   52%   315   38% 
   7 scorpio-143   326   40   40   200   49%   330   35% 
   8 scorpio-142   325   40   40   200   50%   327   37% 
   9 scorpio-133   319   40   40   200   51%   310   37% 
  10 scorpio-135   315   39   39   200   48%   327   43% 
  11 scorpio-137   311   40   40   200   52%   298   34% 
  12 scorpio-136   308   39   39   200   49%   313   37% 
  13 scorpio-139   308   39   39   200   52%   297   37% 
  14 scorpio-140   306   40   40   200   48%   318   34% 
  15 scorpio-138   288   40   40   200   47%   309   37% 
  16 scorpio-131   283   39   39   200   56%   247   39% 
  17 scorpio-132   273   40   40   200   46%   301   36% 
  18 scorpio-125   243   40   40   200   55%   210   32% 
  19 scorpio-128   224   39   39   200   51%   217   37% 
  20 scorpio-129   222   39   39   200   50%   222   40% 
  21 scorpio-130   221   39   39   200   45%   253   41% 
  22 scorpio-124   218   40   40   200   50%   218   34% 
  23 scorpio-127   212   39   39   200   50%   213   39% 
  24 scorpio-121   203   40   40   200   57%   154   35% 
  25 scorpio-126   201   40   40   200   46%   227   35% 
  26 scorpio-123   193   39   39   200   51%   185   38% 
  27 scorpio-118   172   39   39   200   54%   149   41% 
  28 scorpio-120   156   39   39   200   47%   176   38% 
  29 scorpio-122   152   39   39   200   43%   198   38% 
  30 scorpio-117   150   54   54   101   47%   170   42% 
  31 scorpio-119   149   39   39   200   48%   164   42% 
  32 scorpio-17     62   38   38   200   56%    24   50% 
  33 scorpio-16     38   38   38   200   50%    40   50% 
  34 scorpio-14     35   39   39   200   55%     5   41% 
  35 scorpio-15     18   38   38   200   47%    36   47% 
  36 scorpio-18     10   38   38   200   47%    27   50% 
  37 scorpio-115    10   39   39   200   53%   -11   38% 
  38 scorpio-101     8   38   38   200   56%   -25   48% 
  39 scorpio-116     8   55   55   101   50%    11   40% 
  40 scorpio-11      6   27   27   400   52%    -8   42% 
  41 scorpio-21      6   38   38   200   53%   -11   49% 
  42 scorpio-10      3   27   27   400   51%    -2   44% 
  43 scorpio-99      0   39   39   200   55%   -28   43% 
  44 scorpio-20     -6   38   38   200   49%     0   50% 
  45 scorpio-19     -7   37   37   200   49%     2   52% 
  46 scorpio-8      -7   28   28   400   52%   -17   41% 
  47 scorpio-13     -8   38   38   200   48%     8   46% 
  48 scorpio-86    -11   39   39   200   53%   -30   41% 
  49 scorpio-9     -11   27   27   400   49%    -2   43% 
  50 scorpio-22    -17   37   37   200   50%   -16   55% 
  51 scorpio-12    -20   31   31   300   47%     2   45% 
  52 scorpio-109   -21   39   39   200   55%   -53   39% 
  53 scorpio-104   -23   39   39   200   55%   -52   39% 
  54 scorpio-7     -24   27   27   400   51%   -28   45% 
  55 scorpio-112   -24   39   39   200   52%   -37   43% 
  56 scorpio-102   -24   38   38   200   49%   -18   48% 
  57 scorpio-100   -25   38   38   200   45%     4   48% 
  58 scorpio-87    -27   38   38   200   49%   -21   46% 
  59 scorpio-97    -27   39   39   200   53%   -45   42% 
  60 scorpio-114   -30   40   40   200   48%   -15   36% 
  61 scorpio-98    -30   39   39   200   48%   -14   42% 
  62 scorpio-88    -30   38   38   200   51%   -34   50% 
  63 scorpio-55    -32   38   38   200   53%   -50   48% 
  64 scorpio-85    -33   39   39   200   52%   -43   42% 
  65 scorpio-56    -35   38   38   200   51%   -41   48% 
  66 scorpio-111   -35   39   39   200   51%   -37   40% 
  67 scorpio-107   -35   39   39   200   54%   -60   43% 
  68 scorpio-37    -36   37   37   200   51%   -44   60% 
  69 scorpio-24    -37   38   38   200   54%   -61   51% 
  70 scorpio-47    -38   37   37   200   54%   -63   52% 
  71 scorpio-23    -39   37   37   200   48%   -27   53% 
  72 scorpio-113   -39   39   39   200   48%   -27   42% 
  73 scorpio-90    -40   39   39   200   51%   -43   43% 
  74 scorpio-33    -40   38   38   200   54%   -64   48% 
  75 scorpio-89    -42   38   38   200   49%   -35   47% 
  76 scorpio-38    -43   36   36   200   50%   -42   62% 
  77 scorpio-103   -44   39   39   200   47%   -23   42% 
  78 scorpio-36    -44   37   37   200   51%   -48   56% 
  79 scorpio-91    -44   38   38   200   51%   -47   45% 
  80 scorpio-39    -47   37   37   200   50%   -47   59% 
  81 scorpio-6     -48   27   27   400   49%   -45   46% 
  82 scorpio-48    -48   38   38   200   52%   -60   46% 
  83 scorpio-40    -50   37   37   200   53%   -67   54% 
  84 scorpio-43    -50   37   37   200   53%   -66   55% 
  85 scorpio-57    -50   38   38   200   52%   -60   51% 
  86 scorpio-110   -51   39   39   200   47%   -28   37% 
  87 scorpio-34    -51   37   37   200   50%   -50   59% 
  88 scorpio-1     -53   29   29   400   56%   -96   30% 
  89 scorpio-108   -54   38   38   200   46%   -28   48% 
  90 scorpio-92    -55   38   38   200   50%   -54   47% 
  91 scorpio-35    -60   37   37   200   48%   -48   60% 
  92 scorpio-105   -60   38   38   200   47%   -44   45% 
  93 scorpio-95    -61   38   38   200   51%   -66   50% 
  94 scorpio-96    -61   39   39   200   48%   -44   43% 
  95 scorpio-93    -63   39   39   200   50%   -62   40% 
  96 scorpio-44    -64   37   37   200   50%   -62   54% 
  97 scorpio-4     -64   28   28   400   50%   -66   39% 
  98 scorpio-54    -65   37   37   200   49%   -58   54% 
  99 scorpio-106   -65   39   39   200   48%   -48   39% 
 100 scorpio-5     -66   28   28   400   49%   -56   42% 
 101 scorpio-3     -66   28   28   400   51%   -72   39% 
 102 scorpio-42    -68   37   37   200   50%   -69   57% 
 103 scorpio-94    -70   38   38   200   49%   -62   45% 
 104 scorpio-31    -73   37   37   200   52%   -84   57% 
 105 scorpio-59    -74   38   38   200   56%  -112   46% 
 106 scorpio-45    -74   37   37   200   50%   -71   57% 
 107 scorpio-84    -75   39   39   200   49%   -70   44% 
 108 scorpio-32    -77   38   38   200   47%   -57   49% 
 109 scorpio-46    -78   37   37   200   46%   -56   57% 
 110 scorpio-2     -80   28   28   400   47%   -59   38% 
 111 scorpio-49    -83   38   38   200   51%   -87   49% 
 112 scorpio-53    -83   38   38   200   50%   -86   49% 
 113 scorpio-25    -84   38   38   200   47%   -65   50% 
 114 scorpio-58    -86   38   38   200   46%   -62   46% 
 115 scorpio-41    -87   37   37   200   45%   -59   54% 
 116 scorpio-62    -90   39   39   200   58%  -138   39% 
 117 scorpio-30    -91   37   37   200   49%   -85   55% 
 118 scorpio-26    -93   38   38   200   52%  -106   50% 
 119 scorpio-29    -96   38   38   200   54%  -118   47% 
 120 scorpio-52   -107   38   38   200   50%  -104   51% 
 121 scorpio-83   -107   38   38   200   53%  -123   47% 
 122 scorpio-0    -111   42   42   200   42%   -53   24% 
 123 scorpio-51   -125   37   37   200   49%  -116   55% 
 124 scorpio-50   -125   37   37   200   46%  -104   54% 
 125 scorpio-27   -128   38   38   200   48%  -119   50% 
 126 scorpio-63   -134   39   39   200   49%  -131   41% 
 127 scorpio-60   -137   38   38   200   45%  -108   50% 
 128 scorpio-61   -141   39   39   200   46%  -114   44% 
 129 scorpio-71   -141   39   39   200   53%  -157   38% 
 130 scorpio-28   -144   38   38   200   45%  -112   47% 
 131 scorpio-66   -150   38   38   200   52%  -162   44% 
 132 scorpio-72   -155   39   39   200   52%  -164   41% 
 133 scorpio-65   -156   38   38   200   51%  -161   48% 
 134 scorpio-70   -159   38   38   200   51%  -165   48% 
 135 scorpio-67   -168   39   39   200   49%  -160   40% 
 136 scorpio-68   -171   39   39   200   51%  -178   42% 
 137 scorpio-82   -171   39   39   200   46%  -146   45% 
 138 scorpio-64   -172   39   39   200   46%  -145   43% 
 139 scorpio-76   -179   39   39   200   54%  -206   39% 
 140 scorpio-81   -185   39   39   200   53%  -205   42% 
 141 scorpio-73   -186   38   38   200   48%  -176   46% 
 142 scorpio-69   -188   38   38   200   46%  -165   47% 
 143 scorpio-74   -196   39   39   200   50%  -196   38% 
 144 scorpio-79   -203   37   37   200   55%  -233   52% 
 145 scorpio-75   -205   39   39   200   47%  -188   39% 
 146 scorpio-77   -207   40   40   200   50%  -203   35% 
 147 scorpio-78   -228   39   39   200   46%  -205   44% 
 148 scorpio-80   -238   38   38   200   43%  -194   47% 

And with the full inverse of the hessian. The deltas are pretty large with this approach
This makes much more sense indeed and captures the problem with the "incremental elo" approach. Good catch!

Code: Select all

Rank Name          Elo    +    - games score oppo. draws 
   1 scorpio-146   415  632  632   200   56%   379   36% 
   2 scorpio-147   383  635  635   100   45%   415   34% 
   3 scorpio-145   374  630  630   200   50%   375   38% 
   4 scorpio-134   347  605  605   200   55%   317   43% 
   5 scorpio-144   335  627  627   200   48%   350   40% 
   6 scorpio-141   329  621  621   200   52%   315   38% 
   7 scorpio-143   326  625  625   200   49%   330   35% 
   8 scorpio-142   325  623  623   200   50%   327   37% 
   9 scorpio-133   319  603  603   200   51%   310   37% 
  10 scorpio-135   315  607  607   200   48%   327   43% 
  11 scorpio-137   311  611  611   200   52%   298   34% 
  12 scorpio-136   308  609  609   200   49%   313   37% 
  13 scorpio-139   308  616  616   200   52%   297   37% 
  14 scorpio-140   306  618  618   200   48%   318   34% 
  15 scorpio-138   288  614  614   200   47%   309   37% 
  16 scorpio-131   283  599  599   200   56%   247   39% 
  17 scorpio-132   273  601  601   200   46%   301   36% 
  18 scorpio-125   243  587  587   200   55%   210   32% 
  19 scorpio-128   224  593  593   200   51%   217   37% 
  20 scorpio-129   222  595  595   200   50%   222   40% 
  21 scorpio-130   221  597  597   200   45%   253   41% 
  22 scorpio-124   218  585  585   200   50%   218   34% 
  23 scorpio-127   212  591  591   200   50%   213   39% 
  24 scorpio-121   203  580  580   200   57%   154   35% 
  25 scorpio-126   201  589  589   200   46%   227   35% 
  26 scorpio-123   193  584  584   200   51%   185   38% 
  27 scorpio-118   172  575  575   200   54%   149   41% 
  28 scorpio-120   156  578  578   200   47%   176   38% 
  29 scorpio-122   152  582  582   200   43%   198   38% 
  30 scorpio-117   150  574  574   101   47%   170   42% 
  31 scorpio-119   149  577  577   200   48%   164   42% 
  32 scorpio-17     62  339  339   200   56%    24   50% 
  33 scorpio-16     38  342  342   200   50%    40   50% 
  34 scorpio-14     35  349  349   200   55%     5   41% 
  35 scorpio-15     18  345  345   200   47%    36   47% 
  36 scorpio-18     10  336  336   200   47%    27   50% 
  37 scorpio-115    10  298  298   200   53%   -11   38% 
  38 scorpio-101     8  263  263   200   56%   -25   48% 
  39 scorpio-116     8  301  301   101   50%    11   40% 
  40 scorpio-11      6  357  357   400   52%    -8   42% 
  41 scorpio-21      6  326  326   200   53%   -11   49% 
  42 scorpio-10      3  359  359   400   51%    -2   44% 
  43 scorpio-99      0  259  259   200   55%   -28   43% 
  44 scorpio-20     -6  330  330   200   49%     0   50% 
  45 scorpio-19     -7  333  333   200   49%     2   52% 
  46 scorpio-8      -7  363  363   400   52%   -17   41% 
  47 scorpio-13     -8  352  352   200   48%     8   46% 
  48 scorpio-86    -11  239  239   200   53%   -30   41% 
  49 scorpio-9     -11  361  361   400   49%    -2   43% 
  50 scorpio-22    -17  323  323   200   50%   -16   55% 
  51 scorpio-12    -20  356  356   300   47%     2   45% 
  52 scorpio-109   -21  282  282   200   55%   -53   39% 
  53 scorpio-104   -23  270  270   200   55%   -52   39% 
  54 scorpio-7     -24  365  365   400   51%   -28   45% 
  55 scorpio-112   -24  290  290   200   52%   -37   43% 
  56 scorpio-102   -24  265  265   200   49%   -18   48% 
  57 scorpio-100   -25  261  261   200   45%     4   48% 
  58 scorpio-87    -27  240  240   200   49%   -21   46% 
  59 scorpio-97    -27  255  255   200   53%   -45   42% 
  60 scorpio-114   -30  295  295   200   48%   -15   36% 
  61 scorpio-98    -30  257  257   200   48%   -14   42% 
  62 scorpio-88    -30  242  242   200   51%   -34   50% 
  63 scorpio-55    -32  247  247   200   53%   -50   48% 
  64 scorpio-85    -33  238  238   200   52%   -43   42% 
  65 scorpio-56    -35  245  245   200   51%   -41   48% 
  66 scorpio-111   -35  287  287   200   51%   -37   40% 
  67 scorpio-107   -35  277  277   200   54%   -60   43% 
  68 scorpio-37    -36  282  282   200   51%   -44   60% 
  69 scorpio-24    -37  317  317   200   54%   -61   51% 
  70 scorpio-47    -38  260  260   200   54%   -63   52% 
  71 scorpio-23    -39  320  320   200   48%   -27   53% 
  72 scorpio-113   -39  292  292   200   48%   -27   42% 
  73 scorpio-90    -40  244  244   200   51%   -43   43% 
  74 scorpio-33    -40  292  292   200   54%   -64   48% 
  75 scorpio-89    -42  243  243   200   49%   -35   47% 
  76 scorpio-38    -43  280  280   200   50%   -42   62% 
  77 scorpio-103   -44  267  267   200   47%   -23   42% 
  78 scorpio-36    -44  284  284   200   51%   -48   56% 
  79 scorpio-91    -44  245  245   200   51%   -47   45% 
  80 scorpio-39    -47  277  277   200   50%   -47   59% 
  81 scorpio-6     -48  367  367   400   49%   -45   46% 
  82 scorpio-48    -48  259  259   200   52%   -60   46% 
  83 scorpio-40    -50  275  275   200   53%   -67   54% 
  84 scorpio-43    -50  268  268   200   53%   -66   55% 
  85 scorpio-57    -50  244  244   200   52%   -60   51% 
  86 scorpio-110   -51  284  284   200   47%   -28   37% 
  87 scorpio-34    -51  289  289   200   50%   -50   59% 
  88 scorpio-1     -53  376  376   400   56%   -96   30% 
  89 scorpio-108   -54  279  279   200   46%   -28   48% 
  90 scorpio-92    -55  247  247   200   50%   -54   47% 
  91 scorpio-35    -60  287  287   200   48%   -48   60% 
  92 scorpio-105   -60  272  272   200   47%   -44   45% 
  93 scorpio-95    -61  252  252   200   51%   -66   50% 
  94 scorpio-96    -61  253  253   200   48%   -44   43% 
  95 scorpio-93    -63  248  248   200   50%   -62   40% 
  96 scorpio-44    -64  266  266   200   50%   -62   54% 
  97 scorpio-4     -64  370  370   400   50%   -66   39% 
  98 scorpio-54    -65  248  248   200   49%   -58   54% 
  99 scorpio-106   -65  274  274   200   48%   -48   39% 
 100 scorpio-5     -66  368  368   400   49%   -56   42% 
 101 scorpio-3     -66  372  372   400   51%   -72   39% 
 102 scorpio-42    -68  271  271   200   50%   -69   57% 
 103 scorpio-94    -70  250  250   200   49%   -62   45% 
 104 scorpio-31    -73  297  297   200   52%   -84   57% 
 105 scorpio-59    -74  242  242   200   56%  -112   46% 
 106 scorpio-45    -74  264  264   200   50%   -71   57% 
 107 scorpio-84    -75  237  237   200   49%   -70   44% 
 108 scorpio-32    -77  295  295   200   47%   -57   49% 
 109 scorpio-46    -78  262  262   200   46%   -56   57% 
 110 scorpio-2     -80  374  374   400   47%   -59   38% 
 111 scorpio-49    -83  257  257   200   51%   -87   49% 
 112 scorpio-53    -83  250  250   200   50%   -86   49% 
 113 scorpio-25    -84  314  314   200   47%   -65   50% 
 114 scorpio-58    -86  243  243   200   46%   -62   46% 
 115 scorpio-41    -87  273  273   200   45%   -59   54% 
 116 scorpio-62    -90  238  238   200   58%  -138   39% 
 117 scorpio-30    -91  300  300   200   49%   -85   55% 
 118 scorpio-26    -93  311  311   200   52%  -106   50% 
 119 scorpio-29    -96  303  303   200   54%  -118   47% 
 120 scorpio-52   -107  251  251   200   50%  -104   51% 
 121 scorpio-83   -107  237  237   200   53%  -123   47% 
 122 scorpio-0    -111  379  379   200   42%   -53   24% 
 123 scorpio-51   -125  253  253   200   49%  -116   55% 
 124 scorpio-50   -125  255  255   200   46%  -104   54% 
 125 scorpio-27   -128  309  309   200   48%  -119   50% 
 126 scorpio-63   -134  237  237   200   49%  -131   41% 
 127 scorpio-60   -137  240  240   200   45%  -108   50% 
 128 scorpio-61   -141  239  239   200   46%  -114   44% 
 129 scorpio-71   -141  233  233   200   53%  -157   38% 
 130 scorpio-28   -144  306  306   200   45%  -112   47% 
 131 scorpio-66   -150  235  235   200   52%  -162   44% 
 132 scorpio-72   -155  233  233   200   52%  -164   41% 
 133 scorpio-65   -156  236  236   200   51%  -161   48% 
 134 scorpio-70   -159  233  233   200   51%  -165   48% 
 135 scorpio-67   -168  234  234   200   49%  -160   40% 
 136 scorpio-68   -171  234  234   200   51%  -178   42% 
 137 scorpio-82   -171  236  236   200   46%  -146   45% 
 138 scorpio-64   -172  236  236   200   46%  -145   43% 
 139 scorpio-76   -179  233  233   200   54%  -206   39% 
 140 scorpio-81   -185  235  235   200   53%  -205   42% 
 141 scorpio-73   -186  233  233   200   48%  -176   46% 
 142 scorpio-69   -188  234  234   200   46%  -165   47% 
 143 scorpio-74   -196  233  233   200   50%  -196   38% 
 144 scorpio-79   -203  234  234   200   55%  -233   52% 
 145 scorpio-75   -205  233  233   200   47%  -188   39% 
 146 scorpio-77   -207  233  233   200   50%  -203   35% 
 147 scorpio-78   -228  234  234   200   46%  -205   44% 
 148 scorpio-80   -238  234  234   200   43%  -194   47% 

[Daniel Shawul]

There are two more options for computing intervals

Code: Select all

jointdist [p] ... compute intervals from joint distribution
exactdist [p] ... compute intervals assuming exact opponent Elos

With exactdist

Code: Select all

Rank Name          Elo    +    - games score oppo. draws 
   1 scorpio-146   415   40   40   200   56%   379   36% 
   2 scorpio-147   383   56   56   100   45%   415   34% 
   3 scorpio-145   374   40   40   200   50%   375   38% 
   4 scorpio-134   347   39   38   200   55%   317   43% 
   5 scorpio-144   335   39   39   200   48%   350   40% 
   6 scorpio-141   329   39   39   200   52%   315   38% 
   7 scorpio-143   326   40   40   200   49%   330   35% 
   8 scorpio-142   325   40   39   200   50%   327   37% 
   9 scorpio-133   319   40   39   200   51%   310   37% 
  10 scorpio-135   315   39   39   200   48%   327   43% 
  11 scorpio-137   311   40   40   200   52%   298   34% 
  12 scorpio-136   308   39   39   200   49%   313   37% 
  13 scorpio-139   308   39   39   200   52%   297   37% 
  14 scorpio-140   306   40   40   200   48%   318   34% 
  15 scorpio-138   288   39   40   200   47%   309   37% 
  16 scorpio-131   283   40   39   200   56%   247   39% 
  17 scorpio-132   273   40   40   200   46%   301   36% 
  18 scorpio-125   243   40   40   200   55%   210   32% 
  19 scorpio-128   224   39   39   200   51%   217   37% 
  20 scorpio-129   222   39   39   200   50%   222   40% 
  21 scorpio-130   221   39   39   200   45%   253   41% 
  22 scorpio-124   218   40   40   200   50%   218   34% 
  23 scorpio-127   212   39   39   200   50%   213   39% 
  24 scorpio-121   203   40   40   200   57%   154   35% 
  25 scorpio-126   201   40   40   200   46%   227   35% 
  26 scorpio-123   193   39   39   200   51%   185   38% 
  27 scorpio-118   172   39   39   200   54%   149   41% 
  28 scorpio-120   156   39   39   200   47%   176   38% 
  29 scorpio-122   152   39   40   200   43%   198   38% 
  30 scorpio-117   150   54   54   101   47%   170   42% 
  31 scorpio-119   149   39   39   200   48%   164   42% 
  32 scorpio-17     62   38   38   200   56%    24   50% 
  33 scorpio-16     38   38   38   200   50%    40   50% 
  34 scorpio-14     35   39   39   200   55%     5   41% 
  35 scorpio-15     18   38   38   200   47%    36   47% 
  36 scorpio-18     10   38   38   200   47%    27   50% 
  37 scorpio-115    10   39   39   200   53%   -11   38% 
  38 scorpio-101     8   38   38   200   56%   -25   48% 
  39 scorpio-116     8   55   55   101   50%    11   40% 
  40 scorpio-11      6   28   28   400   52%    -8   42% 
  41 scorpio-21      6   38   38   200   53%   -11   49% 
  42 scorpio-10      3   27   27   400   51%    -2   44% 
  43 scorpio-99      0   39   38   200   55%   -28   43% 
  44 scorpio-20     -6   38   38   200   49%     0   50% 
  45 scorpio-19     -7   37   37   200   49%     2   52% 
  46 scorpio-8      -7   28   28   400   52%   -17   41% 
  47 scorpio-13     -8   38   38   200   48%     8   46% 
  48 scorpio-86    -11   39   39   200   53%   -30   41% 
  49 scorpio-9     -11   27   27   400   49%    -2   43% 
  50 scorpio-22    -17   37   37   200   50%   -16   55% 
  51 scorpio-12    -20   31   31   300   47%     2   45% 
  52 scorpio-109   -21   39   39   200   55%   -53   39% 
  53 scorpio-104   -23   39   39   200   55%   -52   39% 
  54 scorpio-7     -24   27   27   400   51%   -28   45% 
  55 scorpio-112   -24   39   38   200   52%   -37   43% 
  56 scorpio-102   -24   38   38   200   49%   -18   48% 
  57 scorpio-100   -25   38   38   200   45%     4   48% 
  58 scorpio-87    -27   38   38   200   49%   -21   46% 
  59 scorpio-97    -27   39   39   200   53%   -45   42% 
  60 scorpio-114   -30   39   40   200   48%   -15   36% 
  61 scorpio-98    -30   39   39   200   48%   -14   42% 
  62 scorpio-88    -30   38   38   200   51%   -34   50% 
  63 scorpio-55    -32   38   38   200   53%   -50   48% 
  64 scorpio-85    -33   39   39   200   52%   -43   42% 
  65 scorpio-56    -35   38   38   200   51%   -41   48% 
  66 scorpio-111   -35   39   39   200   51%   -37   40% 
  67 scorpio-107   -35   39   39   200   54%   -60   43% 
  68 scorpio-37    -36   37   37   200   51%   -44   60% 
  69 scorpio-24    -37   38   37   200   54%   -61   51% 
  70 scorpio-47    -38   38   37   200   54%   -63   52% 
  71 scorpio-23    -39   37   37   200   48%   -27   53% 
  72 scorpio-113   -39   39   39   200   48%   -27   42% 
  73 scorpio-90    -40   39   39   200   51%   -43   43% 
  74 scorpio-33    -40   38   38   200   54%   -64   48% 
  75 scorpio-89    -42   38   38   200   49%   -35   47% 
  76 scorpio-38    -43   36   36   200   50%   -42   62% 
  77 scorpio-103   -44   39   39   200   47%   -23   42% 
  78 scorpio-36    -44   37   37   200   51%   -48   56% 
  79 scorpio-91    -44   38   38   200   51%   -47   45% 
  80 scorpio-39    -47   37   37   200   50%   -47   59% 
  81 scorpio-6     -48   27   27   400   49%   -45   46% 
  82 scorpio-48    -48   38   38   200   52%   -60   46% 
  83 scorpio-40    -50   37   37   200   53%   -67   54% 
  84 scorpio-43    -50   37   37   200   53%   -66   55% 
  85 scorpio-57    -50   38   38   200   52%   -60   51% 
  86 scorpio-110   -51   39   40   200   47%   -28   37% 
  87 scorpio-34    -51   37   37   200   50%   -50   59% 
  88 scorpio-1     -53   29   29   400   56%   -96   30% 
  89 scorpio-108   -54   38   38   200   46%   -28   48% 
  90 scorpio-92    -55   38   38   200   50%   -54   47% 
  91 scorpio-35    -60   37   37   200   48%   -48   60% 
  92 scorpio-105   -60   38   38   200   47%   -44   45% 
  93 scorpio-95    -61   38   38   200   51%   -66   50% 
  94 scorpio-96    -61   38   39   200   48%   -44   43% 
  95 scorpio-93    -63   39   39   200   50%   -62   40% 
  96 scorpio-44    -64   37   37   200   50%   -62   54% 
  97 scorpio-4     -64   28   28   400   50%   -66   39% 
  98 scorpio-54    -65   37   37   200   49%   -58   54% 
  99 scorpio-106   -65   39   39   200   48%   -48   39% 
 100 scorpio-5     -66   28   28   400   49%   -56   42% 
 101 scorpio-3     -66   28   28   400   51%   -72   39% 
 102 scorpio-42    -68   37   37   200   50%   -69   57% 
 103 scorpio-94    -70   38   38   200   49%   -62   45% 
 104 scorpio-31    -73   37   37   200   52%   -84   57% 
 105 scorpio-59    -74   38   38   200   56%  -112   46% 
 106 scorpio-45    -74   37   37   200   50%   -71   57% 
 107 scorpio-84    -75   38   38   200   49%   -70   44% 
 108 scorpio-32    -77   38   38   200   47%   -57   49% 
 109 scorpio-46    -78   37   37   200   46%   -56   57% 
 110 scorpio-2     -80   28   28   400   47%   -59   38% 
 111 scorpio-49    -83   38   38   200   51%   -87   49% 
 112 scorpio-53    -83   38   38   200   50%   -86   49% 
 113 scorpio-25    -84   38   38   200   47%   -65   50% 
 114 scorpio-58    -86   38   38   200   46%   -62   46% 
 115 scorpio-41    -87   37   37   200   45%   -59   54% 
 116 scorpio-62    -90   39   39   200   58%  -138   39% 
 117 scorpio-30    -91   37   37   200   49%   -85   55% 
 118 scorpio-26    -93   38   38   200   52%  -106   50% 
 119 scorpio-29    -96   38   38   200   54%  -118   47% 
 120 scorpio-52   -107   38   38   200   50%  -104   51% 
 121 scorpio-83   -107   38   38   200   53%  -123   47% 
 122 scorpio-0    -111   41   42   200   42%   -53   24% 
 123 scorpio-51   -125   37   37   200   49%  -116   55% 
 124 scorpio-50   -125   37   37   200   46%  -104   54% 
 125 scorpio-27   -128   38   38   200   48%  -119   50% 
 126 scorpio-63   -134   39   39   200   49%  -131   41% 
 127 scorpio-60   -137   38   38   200   45%  -108   50% 
 128 scorpio-61   -141   38   39   200   46%  -114   44% 
 129 scorpio-71   -141   39   39   200   53%  -157   38% 
 130 scorpio-28   -144   38   38   200   45%  -112   47% 
 131 scorpio-66   -150   38   38   200   52%  -162   44% 
 132 scorpio-72   -155   39   39   200   52%  -164   41% 
 133 scorpio-65   -156   38   38   200   51%  -161   48% 
 134 scorpio-70   -159   38   38   200   51%  -165   48% 
 135 scorpio-67   -168   39   39   200   49%  -160   40% 
 136 scorpio-68   -171   39   39   200   51%  -178   42% 
 137 scorpio-82   -171   38   39   200   46%  -146   45% 
 138 scorpio-64   -172   38   39   200   46%  -145   43% 
 139 scorpio-76   -179   39   39   200   54%  -206   39% 
 140 scorpio-81   -185   39   39   200   53%  -205   42% 
 141 scorpio-73   -186   38   38   200   48%  -176   46% 
 142 scorpio-69   -188   38   38   200   46%  -165   47% 
 143 scorpio-74   -196   39   39   200   50%  -196   38% 
 144 scorpio-79   -203   37   37   200   55%  -233   52% 
 145 scorpio-75   -205   39   39   200   47%  -188   39% 
 146 scorpio-77   -207   40   40   200   50%  -203   35% 
 147 scorpio-78   -228   38   39   200   46%  -205   44% 
 148 scorpio-80   -238   38   38   200   43%  -194   47% 
 

jointdist takes a lot of time. Will post again if it finishes.

[dragontamer5788]

2 scorpio-147 383 56 56 100 45% 415 34%

So if I'm reading this correctly: #147 finished with 383 Elo (+/- 56) and played 100 matches. But all of its matches were against #146 (the only Elo score of 415 in the entire chart). That seems like a recipe for poor statistics.

How did you determine which networks would match up against other networks? I would expect a tournament design to start off with random matches, followed up by maybe "focused" matches with a particular partner to maybe solidify the scores.

EDIT: A score of +/- 600 Elo means that your estimated probability-of-win is somewhere between 3% and 97%. Obviously, these error bounds are huge. I would expect better error-bounds after 80,000 games.