Round Robin simulations program

[michiguel]

RRSIM

https://sites.google.com/site/gaviotach ... eding_edge

Readme flle

Code: Select all

# RRSIM

Simulation of round robin tournaments.

Model based on the program Ordo

### Compilation
Program can be compile and installed in GNU/Linux with

`make`

`make install`

### Quick Example

`rrsim -i input_example.csv -s100000`

Other switches provide the ability to determine white advantage, draw rate, whether colors are reversed, and how many legs the round-robin tournament has (see below).

### Input file

Input files are comma separated values. They are text files with two colums.
The first one contains the name of the engine with quotes, and in the second one, its rating.
For instance, it should look like this:

```
"Engine A", 3200
"Engine B", 3100
"Engine C", 3050
"Engine D", 3000
"Engine E", 2950
"Engine F", 2900
"Engine G", 2700
"Engine H", 2700
"Engine I", 2200
"Engine J", 2200
"Engine K", 2200
```

### Help reported by switch -h

```
Program to simulate round robin tournaments

quick example: rrsim -i input_example.csv -w30 -d30 -s100 -p simulated.pgn
  - Processes input_example.csv 
  - White advantage is 30 points, draw rate is 30%, simulated 100 times
  - Games simulated are saved in simulated.pgn

usage: rrsim [-OPTION]
 -h        print this help
 -v        print version number and exit
 -L        display the license information
 -q        quiet (no output except error messages)
 -i <file> input file, with comma separated value format
 -p <file> output file, containing all simulated games &#40;optional&#41;
 -w <num>  white advantage &#40;rating points&#41;, default = 0.0
 -d <num>  draw rate for equal opponents (%), default = 50.0
 -s <num>  number of simulation repeats, default = 1
 -r        starts with reversed colors, &#40;optional&#41;
 -l        round robing legs, default = 1

Copyright &#40;c&#41; 2015 Miguel A. Ballicora
Thereis NO WARRANTY of any kind
```

Output example (work in progress)

Code: Select all

0   10   20   30   40   50   60   70   80   90   100 (%)
|----|----|----|----|----|----|----|----|----|----|
[*************************************************]

Tournament engines = 11
Tournament boards = 5
Tournament games/leg = 55
Tournament rounds/leg = 11
Tournament legs = 1
Tournament total games = 55
Tournament total rounds = 11
Simulations = 100000
Total games = 5500000
draw rate (equal strength) = 50.0%
White advantage = 0.0

First Engine (Engine A: 3200.0) Stats:
won    = 56411
shared = 17373
loss   = 26216
total  = 100000
won outright % = 56.4
won shared   % = 17.4

As useful feature for testing rating software is that with the switch -p, you can save all simulated game results in pgn format.

Miguel

[Laskos]

RRSIM

https://sites.google.com/site/gaviotach ... eding_edge

Readme flle

Code: Select all

# RRSIM

Simulation of round robin tournaments.

Model based on the program Ordo

### Compilation
Program can be compile and installed in GNU/Linux with

`make`

`make install`

### Quick Example

`rrsim -i input_example.csv -s100000`

Other switches provide the ability to determine white advantage, draw rate, whether colors are reversed, and how many legs the round-robin tournament has (see below).

### Input file

Input files are comma separated values. They are text files with two colums.
The first one contains the name of the engine with quotes, and in the second one, its rating.
For instance, it should look like this:

```
"Engine A", 3200
"Engine B", 3100
"Engine C", 3050
"Engine D", 3000
"Engine E", 2950
"Engine F", 2900
"Engine G", 2700
"Engine H", 2700
"Engine I", 2200
"Engine J", 2200
"Engine K", 2200
```

### Help reported by switch -h

```
Program to simulate round robin tournaments

quick example: rrsim -i input_example.csv -w30 -d30 -s100 -p simulated.pgn
  - Processes input_example.csv 
  - White advantage is 30 points, draw rate is 30%, simulated 100 times
  - Games simulated are saved in simulated.pgn

usage: rrsim [-OPTION]
 -h        print this help
 -v        print version number and exit
 -L        display the license information
 -q        quiet (no output except error messages)
 -i <file> input file, with comma separated value format
 -p <file> output file, containing all simulated games &#40;optional&#41;
 -w <num>  white advantage &#40;rating points&#41;, default = 0.0
 -d <num>  draw rate for equal opponents (%), default = 50.0
 -s <num>  number of simulation repeats, default = 1
 -r        starts with reversed colors, &#40;optional&#41;
 -l        round robing legs, default = 1

Copyright &#40;c&#41; 2015 Miguel A. Ballicora
Thereis NO WARRANTY of any kind
```

Output example (work in progress)

Code: Select all

0   10   20   30   40   50   60   70   80   90   100 (%)
|----|----|----|----|----|----|----|----|----|----|
[*************************************************]

Tournament engines = 11
Tournament boards = 5
Tournament games/leg = 55
Tournament rounds/leg = 11
Tournament legs = 1
Tournament total games = 55
Tournament total rounds = 11
Simulations = 100000
Total games = 5500000
draw rate (equal strength) = 50.0%
White advantage = 0.0

First Engine (Engine A: 3200.0) Stats:
won    = 56411
shared = 17373
loss   = 26216
total  = 100000
won outright % = 56.4
won shared   % = 17.4

As useful feature for testing rating software is that with the switch -p, you can save all simulated game results in pgn format.

Miguel

Thanks Miguel, works smoothly. What are the draw values in case of non-equal opponents?

There are two important competing models: Rao-Kupper used in Bayeselo, that assumes 1 win + 1 loss = 1 draw, but doesn't fit well empirical data, and Davidson, which assumes 1 win + 1 loss = 2 draws, and fits better empirical data.

Code: Select all

Rao-Kupper: 
d = C*w*(1 - w - d) 
d -> (C*w - C*w^2)/(1 + C*w)

Code: Select all

Davidson: 
d^2 = C*w*(1 - w - d) 
d -> 1/2 (-C*w + Sqrt[C]*Sqrt[w]*Sqrt[4 - 4 w + C*w])

Where d=draw ratio, win=win ratio, C is an empirical constant to be determined. Did you use one of those?

[michiguel]

RRSIM

https://sites.google.com/site/gaviotach ... eding_edge

Readme flle

Code: Select all

# RRSIM

Simulation of round robin tournaments.

Model based on the program Ordo

### Compilation
Program can be compile and installed in GNU/Linux with

`make`

`make install`

### Quick Example

`rrsim -i input_example.csv -s100000`

Other switches provide the ability to determine white advantage, draw rate, whether colors are reversed, and how many legs the round-robin tournament has (see below).

### Input file

Input files are comma separated values. They are text files with two colums.
The first one contains the name of the engine with quotes, and in the second one, its rating.
For instance, it should look like this:

```
"Engine A", 3200
"Engine B", 3100
"Engine C", 3050
"Engine D", 3000
"Engine E", 2950
"Engine F", 2900
"Engine G", 2700
"Engine H", 2700
"Engine I", 2200
"Engine J", 2200
"Engine K", 2200
```

### Help reported by switch -h

```
Program to simulate round robin tournaments

quick example: rrsim -i input_example.csv -w30 -d30 -s100 -p simulated.pgn
  - Processes input_example.csv 
  - White advantage is 30 points, draw rate is 30%, simulated 100 times
  - Games simulated are saved in simulated.pgn

usage: rrsim [-OPTION]
 -h        print this help
 -v        print version number and exit
 -L        display the license information
 -q        quiet (no output except error messages)
 -i <file> input file, with comma separated value format
 -p <file> output file, containing all simulated games &#40;optional&#41;
 -w <num>  white advantage &#40;rating points&#41;, default = 0.0
 -d <num>  draw rate for equal opponents (%), default = 50.0
 -s <num>  number of simulation repeats, default = 1
 -r        starts with reversed colors, &#40;optional&#41;
 -l        round robing legs, default = 1

Copyright &#40;c&#41; 2015 Miguel A. Ballicora
Thereis NO WARRANTY of any kind
```

Output example (work in progress)

Code: Select all

0   10   20   30   40   50   60   70   80   90   100 (%)
|----|----|----|----|----|----|----|----|----|----|
[*************************************************]

Tournament engines = 11
Tournament boards = 5
Tournament games/leg = 55
Tournament rounds/leg = 11
Tournament legs = 1
Tournament total games = 55
Tournament total rounds = 11
Simulations = 100000
Total games = 5500000
draw rate (equal strength) = 50.0%
White advantage = 0.0

First Engine (Engine A: 3200.0) Stats:
won    = 56411
shared = 17373
loss   = 26216
total  = 100000
won outright % = 56.4
won shared   % = 17.4

As useful feature for testing rating software is that with the switch -p, you can save all simulated game results in pgn format.

Miguel

Thanks Miguel, works smoothly. What are the draw values in case of non-equal opponents?

There are two important competing models: Rao-Kupper used in Bayeselo, that assumes 1 win + 1 loss = 1 draw, but doesn't fit well empirical data, and Davidson, which assumes 1 win + 1 loss = 2 draws, and fits better empirical data.

Code: Select all

Rao-Kupper: 
d = C*w*(1 - w - d) 
d -> (C*w - C*w^2)/(1 + C*w)

Code: Select all

Davidson: 
d^2 = C*w*(1 - w - d) 
d -> 1/2 (-C*w + Sqrt[C]*Sqrt[w]*Sqrt[4 - 4 w + C*w])

Where d=draw ratio, win=win ratio, C is an empirical constant to be determined. Did you use one of those?

I did my own derivation assuming strength is like energy levels:
https://sites.google.com/site/gaviotach ... rdo-readme

But I see that it converges to Davidson (see eq 29 in the readme). In Ordo's case (and in the simulation) the parameter C equals

C = [2 * Deq / (1 - Deq) ]^2

Where Deq is the draw rate between two opponents of equal strength.

I may need to say this in the readme and cite Davidson. Do you have a link? I guess I can find it.

Miguel

[Adam Hair]

Davidson's paper: http://stat.fsu.edu/techreports/M169.pdf

[Laskos]

Thanks Miguel, works smoothly. What are the draw values in case of non-equal opponents?

There are two important competing models: Rao-Kupper used in Bayeselo, that assumes 1 win + 1 loss = 1 draw, but doesn't fit well empirical data, and Davidson, which assumes 1 win + 1 loss = 2 draws, and fits better empirical data.

Code: Select all

Rao-Kupper: 
d = C*w*(1 - w - d) 
d -> (C*w - C*w^2)/(1 + C*w)

Code: Select all

Davidson: 
d^2 = C*w*(1 - w - d) 
d -> 1/2 (-C*w + Sqrt[C]*Sqrt[w]*Sqrt[4 - 4 w + C*w])

Where d=draw ratio, win=win ratio, C is an empirical constant to be determined. Did you use one of those?

I did my own derivation assuming strength is like energy levels:
https://sites.google.com/site/gaviotach ... rdo-readme

But I see that it converges to Davidson (see eq 29 in the readme). In Ordo's case (and in the simulation) the parameter C equals

C = [2 * Deq / (1 - Deq) ]^2

Where Deq is the draw rate between two opponents of equal strength.

I may need to say this in the readme and cite Davidson. Do you have a link? I guess I can find it.

Miguel

I think that is the paper:

Code: Select all

Davidson, R. R. (1970). On extending the Bradley-Terry model to accommodate ties in paired comparison experiments. Journal of the American Statistical Association, 65(329):317–328.

Also, I have the following useful link:
https://dl.dropboxusercontent.com/u/552 ... tcomes.pdf

I see that you use a model converging to the Davidson one, which is fine.

[Adam Hair]

Code: Select all

Davidson: 
d^2 = C*w*(1 - w - d) 
d -> 1/2 (-C*w + Sqrt[C]*Sqrt[w]*Sqrt[4 - 4 w + C*w])

Where d=draw ratio, win=win ratio, C is an empirical constant to be determined. Did you use one of those?

As a side note, Miguel may have found an improvement over Davidson's model. If we replace the constant term Deq in Miguel's equation with the linear term mW+b, where m and b reflect how the draw rate changes in the database, a better fit occurs.

The following graphs come from an experiment I performed last year with Gaviota. The data represent the draw rate versus win rate of Gaviota 64k (64k nodes per move) versus Gaviota 1k through Gaviota 256k. Each graph compares a draw model to the data.

Rao Kupper



Davidson/Ordo



Davidson/Ordo with linear correction

[michiguel]

Thanks Miguel, works smoothly. What are the draw values in case of non-equal opponents?

There are two important competing models: Rao-Kupper used in Bayeselo, that assumes 1 win + 1 loss = 1 draw, but doesn't fit well empirical data, and Davidson, which assumes 1 win + 1 loss = 2 draws, and fits better empirical data.

Code: Select all

Rao-Kupper: 
d = C*w*(1 - w - d) 
d -> (C*w - C*w^2)/(1 + C*w)

Code: Select all

Davidson: 
d^2 = C*w*(1 - w - d) 
d -> 1/2 (-C*w + Sqrt[C]*Sqrt[w]*Sqrt[4 - 4 w + C*w])

Where d=draw ratio, win=win ratio, C is an empirical constant to be determined. Did you use one of those?

I did my own derivation assuming strength is like energy levels:
https://sites.google.com/site/gaviotach ... rdo-readme

But I see that it converges to Davidson (see eq 29 in the readme). In Ordo's case (and in the simulation) the parameter C equals

C = [2 * Deq / (1 - Deq) ]^2

Where Deq is the draw rate between two opponents of equal strength.

I may need to say this in the readme and cite Davidson. Do you have a link? I guess I can find it.

Miguel

I think that is the paper:

Code: Select all

Davidson, R. R. (1970). On extending the Bradley-Terry model to accommodate ties in paired comparison experiments. Journal of the American Statistical Association, 65(329):317–328.

Also, I have the following useful link:
https://dl.dropboxusercontent.com/u/552 ... tcomes.pdf

I see that you use a model converging to the Davidson one, which is fine.

Thanks Adam & Kai, I incorporated those links in the Ordo's MANUAL.pdf file

Miguel

[Laskos]

Thanks Adam & Kai, I incorporated those links in the Ordo's MANUAL.pdf file

Miguel

(29) in the Manual has a typo, the power is 1/2, not (-2).

[michiguel]

Thanks Adam & Kai, I incorporated those links in the Ordo's MANUAL.pdf file

Miguel

(29) in the Manual has a typo, the power is 1/2, not (-2).

Ouch!

Thanks, I fixed it.
Miguel

[Ajedrecista]

Hello Miguel:

This is what I needed for tournaments such as TCEC or my beloved Linares... it it comes back.

I think your programme is very good and fast but it only shows the winning possibilities of the first player of the list in the .csv file. Other option could be adding the option of write the winning probabilities of all the players in an output notepad.

Just testing it, I took the example of the 2007 World Chess Championship (which I enjoyed a lot), and fixed the following conditions:

Code: Select all

~\rrsim-v0.5-win>rrsim-win32.exe -i input_example.csv -w 40 -d 65 -s 1000000 -l 2
Simulations: 1000000

0   10   20   30   40   50   60   70   80   90   100 (%)
|----|----|----|----|----|----|----|----|----|----|
[*************************************************]

Tournament engines = 8
Tournament boards = 4
Tournament games/leg = 28
Tournament rounds/leg = 7
Tournament legs = 2
Tournament total games = 56
Tournament total rounds = 14
Simulations = 1000000
Total games = 56000000
draw rate (equal strength) = 65.0%
White advantage = 40.0

Code: Select all

"Anand", 2792
"Kramnik", 2769
"Gelfand", 2733
"Leko", 2751
"Svidler", 2735
"Morozevich", 2758
"Aronian", 2750
"Grischuk", 2726

Then, changing the order of the list to get the stats of all players:

Code: Select all

First Engine (Anand: 2792.0) Stats:
won    = 267831
shared = 130406
loss   = 601763
total  = 1000000
won outright % = 26.8
won shared   % = 13.0

------------------------

First Engine (Kramnik: 2769.0) Stats:
won    = 136175
shared = 94028
loss   = 769797
total  = 1000000
won outright % = 13.6
won shared   % = 9.4

------------------------

First Engine (Gelfand: 2733.0) Stats:
won    = 39014
shared = 42004
loss   = 918982
total  = 1000000
won outright % = 3.9
won shared   % = 4.2

------------------------

First Engine (Leko: 2751.0) Stats:
won    = 75164
shared = 65492
loss   = 859344
total  = 1000000
won outright % = 7.5
won shared   % = 6.5

------------------------

First Engine (Svidler: 2735.0) Stats:
won    = 42415
shared = 44124
loss   = 913461
total  = 1000000
won outright % = 4.2
won shared   % = 4.4

------------------------

First Engine (Morozevich: 2758.0) Stats:
won    = 94842
shared = 76585
loss   = 828573
total  = 1000000
won outright % = 9.5
won shared   % = 7.7

------------------------

First Engine (Aronian: 2750.0) Stats:
won    = 72652
shared = 64093
loss   = 863255
total  = 1000000
won outright % = 7.3
won shared   % = 6.4

------------------------

First Engine (Grischuk: 2726.0) Stats:
won    = 29764
shared = 34265
loss   = 935971
total  = 1000000
won outright % = 3.0
won shared   % = 3.4

Code: Select all

Total of won outright: 267831 + 136175 + 39014 + 75164 + 42415 + 94842 + 72652 + 29764 = 757857
Total of won shared:   130406 +  94028 + 42004 + 65492 + 44124 + 76585 + 64093 + 34265 = 550997

With my settings and after 1e+6 simulations, almost 75.8% of cases end with an only winner. Since (won outright) + (won shared)/2 > simulations, I guess that there are some cases where the shares are of more than two players. I would have said the same just seen that won shared is an odd number.

Once again, thanks for your great job!

Regards from Spain.

Ajedrecista.