michiguel wrote:
I have the feeling that using books may increase white advantage. Anyway, simulations with 30 elo of white advantage are below.
Tie breaks are not applied. Those cases are the ones listed as "shared". When there was only one winner is listed as "outright".
I can try to implement some SB rules.
===========
Color when 1 plays white against 2
Total engines = 11
Total games = 55
Total rounds = 11
Total boards = 5
Total cycles = 1000000
draw rate (equal strength) = 64.0%
White advantage = 30.0
rating[0]=3200
rating[1]=3100
rating[2]=3050
rating[3]=3000
rating[4]=2950
rating[5]=2900
rating[6]=2700
rating[7]=2700
rating[8]=2200
rating[9]=2200
rating[10]=2200
won = 622003
shared = 177919
loss = 200078
total = 1000000
won outright % = 62.2 <===============
won shared % = 17.8
========
Reversed colors
Total engines = 11
Total games = 55
Total rounds = 11
Total boards = 5
Total cycles = 1000000
draw rate (equal strength) = 64.0%
White advantage = 30.0
rating[0]=3200
rating[1]=3100
rating[2]=3050
rating[3]=3000
rating[4]=2950
rating[5]=2900
rating[6]=2700
rating[7]=2700
rating[8]=2200
rating[9]=2200
rating[10]=2200
won = 586411
shared = 186268
loss = 227321
total = 1000000
won outright % = 58.6 <===============
won shared % = 18.6
I used drawelo of 200, which gives lower draw rate than your 64% for equal opponents. 64% seems a bit high, it's valid only for several top engines, the rest have lower draw rates. Can you use 56%? Probably my drawelo of 200 is a bit too low, though.
I used 64% just to compare with Peter's. Just to make it clear, this is between equal opponents. Between opponents of different strength scales down automatically.
One of the limitations of Ordo (and any other rating software for that matter) is that the draw rate (between equal opponents) is assumed constant throughout the rating spectrum, and this is not true. So, an average needs to be used (the model will be improved when I work with the wilo model).
Here are several combinations (reversed colors, #2 plays white against #1)
draw rate (equal strength) = 56.0%
White advantage = 30.0
won = 564497
shared = 179396
loss = 256107
total = 1000000
won outright % = 56.4
won shared % = 17.9
draw rate (equal strength) = 50.0%
White advantage = 30.0
won = 548826
shared = 173783
loss = 277391
total = 1000000
won outright % = 54.9
won shared % = 17.4
draw rate (equal strength) = 40.0%
White advantage = 30.0
won = 524417
shared = 164144
loss = 311439
total = 1000000
won outright % = 52.4
won shared % = 16.4
Milos wrote:
In ICGA it's not SB but 2 games match with reversed colors and if still tied, a sudden death game with 20% more time for white.
No it is not. It is either a Swiss or a double round robin. There is only a tie break IF 2 engines are tied at the end of the tournament.
It is never Swiss since there are not enough participants (last time it was Swiss was like 9 years ago, and before that 11 years ago).
In more than 80% of the cases if there is a tie it is between 2 engines only.
michiguel wrote:
I have the feeling that using books may increase white advantage. Anyway, simulations with 30 elo of white advantage are below.
Tie breaks are not applied. Those cases are the ones listed as "shared". When there was only one winner is listed as "outright".
I can try to implement some SB rules.
===========
Color when 1 plays white against 2
Total engines = 11
Total games = 55
Total rounds = 11
Total boards = 5
Total cycles = 1000000
draw rate (equal strength) = 64.0%
White advantage = 30.0
rating[0]=3200
rating[1]=3100
rating[2]=3050
rating[3]=3000
rating[4]=2950
rating[5]=2900
rating[6]=2700
rating[7]=2700
rating[8]=2200
rating[9]=2200
rating[10]=2200
won = 622003
shared = 177919
loss = 200078
total = 1000000
won outright % = 62.2 <===============
won shared % = 17.8
========
Reversed colors
Total engines = 11
Total games = 55
Total rounds = 11
Total boards = 5
Total cycles = 1000000
draw rate (equal strength) = 64.0%
White advantage = 30.0
rating[0]=3200
rating[1]=3100
rating[2]=3050
rating[3]=3000
rating[4]=2950
rating[5]=2900
rating[6]=2700
rating[7]=2700
rating[8]=2200
rating[9]=2200
rating[10]=2200
won = 586411
shared = 186268
loss = 227321
total = 1000000
won outright % = 58.6 <===============
won shared % = 18.6
I used drawelo of 200, which gives lower draw rate than your 64% for equal opponents. 64% seems a bit high, it's valid only for several top engines, the rest have lower draw rates. Can you use 56%? Probably my drawelo of 200 is a bit too low, though.
I used 64% just to compare with Peter's. Just to make it clear, this is between equal opponents. Between opponents of different strength scales down automatically.
One of the limitations of Ordo (and any other rating software for that matter) is that the draw rate (between equal opponents) is assumed constant throughout the rating spectrum, and this is not true. So, an average needs to be used (the model will be improved when I work with the wilo model).
Here are several combinations (reversed colors, #2 plays white against #1)
draw rate (equal strength) = 56.0%
White advantage = 30.0
won = 564497
shared = 179396
loss = 256107
total = 1000000
won outright % = 56.4
won shared % = 17.9
draw rate (equal strength) = 50.0%
White advantage = 30.0
won = 548826
shared = 173783
loss = 277391
total = 1000000
won outright % = 54.9
won shared % = 17.4
draw rate (equal strength) = 40.0%
White advantage = 30.0
won = 524417
shared = 164144
loss = 311439
total = 1000000
won outright % = 52.4
won shared % = 16.4
Thanks, that is very close to my results (52% draw rate for drawelo 200 and equal opponents). In fact I used a hacked in 5 minutes simulator for TCEC, where the drawelo_var was a variable function of Elos of opponents as drawelo*Elo/3200, with SF and Komodo Elo at 3200, and drawelo of 260. This is a gross approximation to take into account the strength of engines. When hacking the sim, I set drawelo 200 for WCCC, but inadvertently kept the dependence on Elos.
Then, in case of ties, I used random assignment of places, say 2 engines are tied, one engine has 50% of a win in that run, 50% the other. Didn't use colors.
Milos wrote:
In ICGA it's not SB but 2 games match with reversed colors and if still tied, a sudden death game with 20% more time for white.
No it is not. It is either a Swiss or a double round robin. There is only a tie break IF 2 engines are tied at the end of the tournament.
It is never Swiss since there are not enough participants (last time it was Swiss was like 9 years ago, and before that 11 years ago).
In more than 80% of the cases if there is a tie it is between 2 engines only.
Learn your facts before commenting!
My facts are correct as I was at all the tournaments. There have been some all play alls without reversed colours. But the main fact is there is not a play off if the 2 games against 1 opponent are both draws.
Milos wrote:
In ICGA it's not SB but 2 games match with reversed colors and if still tied, a sudden death game with 20% more time for white.
No it is not. It is either a Swiss or a double round robin. There is only a tie break IF 2 engines are tied at the end of the tournament.
It is never Swiss since there are not enough participants (last time it was Swiss was like 9 years ago, and before that 11 years ago).
In more than 80% of the cases if there is a tie it is between 2 engines only.
Learn your facts before commenting!
My facts are correct as I was at all the tournaments. There have been some all play alls without reversed colours. But the main fact is there is not a play off if the 2 games against 1 opponent are both draws.
Double round-robin was played only at the last WCCC since there was an inherent lack of participants. In the past it was always all-play-alls without reversed colors. Whenever 2 engines were tied for the first place, there was always a play-off
The example I gave here are exactly the most realistic ones (assuming a hypothetical case of SF participating), i.e. 11 participants meaning all-plays-all, no round-robin, and if two engines are tied for the first place there is always a play-off.
Own books. A championship should be the full effort of one opponent against the other, and that includes opening preparation. Engines should use their own books. Even better would be tuning for specific opponents, since opponent modeling is still an unsolved problem. Although problematic for the long format of the TCEC, it might be more workable for a shorter tournament. Having one stage be bookless this year was a good compromise.
I won't comment books, but my personal opinion is that books have nothing to do with engine programming, and contrary to what ICGA (which claims to be the competition of programmers) is doing, TCEC should actually be the competition of best engine programming.
There is an entire meta-game of selecting the right opening and best lines to exploit the known weaknesses of an opponent and to play to your own strengths. This is an area of deep concern to human players, which as far as I know has never been tackled by computer programs. Why not? It is a very interesting opponent modeling problem! In fact, it is one of the few areas in which computer chess can still serve as an AI drosophila.
No adjudication. The TCEC game ending rules are OK to shorten a very lengthy format, but still have the potential for introducing error. A real championship should play to the bitter end.
This is pretty unreasonable. Why in human chess offering draws and resigning is allowed while in computer chess you would prevent it???
In all the games during all TCEC seasons there might have been 2 or 3 games in total where results was wrongly adjudicated, a percentage much lower than in case of human chess.
There is a difference: in this case the opponents themselves are using their own logic to resign, offer draws, and accept draws. To that I have no objection (but does the UCI protocol allow draw negotiation? I forget...) What I am against is a third-party making the decisions (TCEC interpretation of engine scores).
Laskos wrote:Rating distortion is a known issue with assumption 1 win + 1 loss = 1 draw, but for simplicity I kept this draw model. Could you do simulations with the following elos I used to get around 60% for SF:
Here I included 3 weaker engines, which basically lose a everything against stronger ones. They are usual occurrences in WCCC. Drawelo=200, white_advantage=0.
Laskos wrote:Rating distortion is a known issue with assumption 1 win + 1 loss = 1 draw, but for simplicity I kept this draw model. Could you do simulations with the following elos I used to get around 60% for SF:
Here I included 3 weaker engines, which basically lose a everything against stronger ones. They are usual occurrences in WCCC. Drawelo=200, white_advantage=0.
Only the first tie-break rule has been implemented.
Thanks Peter, that's close to my results, we differ on tie rules and my draw model was a bit modified drawelo adjusted for absolute strength (in a bit clumsy way).
IanO wrote:There is a difference: in this case the opponents themselves are using their own logic to resign, offer draws, and accept draws. To that I have no objection (but does the UCI protocol allow draw negotiation? I forget...) What I am against is a third-party making the decisions (TCEC interpretation of engine scores).
There is no obligation to use UCI, right? It is a choice of the participants. If my engine is defective in other respects, say it cannot find checkmate in KQK, that would be no reason to alter the rules of Chess and declare KQK won by the 'baring rule'. I should bloody well fix my engine...
Milos wrote:
In ICGA it's not SB but 2 games match with reversed colors and if still tied, a sudden death game with 20% more time for white.
No it is not. It is either a Swiss or a double round robin. There is only a tie break IF 2 engines are tied at the end of the tournament.
It is never Swiss since there are not enough participants (last time it was Swiss was like 9 years ago, and before that 11 years ago).
In more than 80% of the cases if there is a tie it is between 2 engines only.
Learn your facts before commenting!
My facts are correct as I was at all the tournaments. There have been some all play alls without reversed colours. But the main fact is there is not a play off if the 2 games against 1 opponent are both draws.
Double round-robin was played only at the last WCCC since there was an inherent lack of participants. In the past it was always all-play-alls without reversed colors. Whenever 2 engines were tied for the first place, there was always a play-off
The example I gave here are exactly the most realistic ones (assuming a hypothetical case of SF participating), i.e. 11 participants meaning all-plays-all, no round-robin, and if two engines are tied for the first place there is always a play-off.
You stated that there is a playoff if the 2 games between opponents are drawn I guess that is not what you meant.
The rules also say there is a playoff if people are tied for 2nd and 3rd as Hiarcs and pandix were once. gyula and I could not be bothered to play for 3rd so we agreed to share with the ICGAs blessing.
