Pawn Advantage, Win Percentage, and Elo
Posted: Sun Apr 15, 2012 8:29 pm
Approximately 4 years ago, Pradu Kannan and Sune Fischer conducted a study to determine the relationship between pawn advantage, win percentage, and Elo (can be found at the Chess Programming Wiki). As a side product of some data mining I am doing, I can offer some additional confirmation of their findings.
I collected over 900,000 long time control engine vs engine games. Then, using Norm Pollock's utilities, I filtered the games so that each opponent was greater than 2700 Elo (CCRL scale) and, for each engine pairing, the two opponents were within 50 Elo of each other. This produced a database of 343,789 games to study.
Using ChessDB (a branch of Scid by Dr. David Kirby), I determined the winning percentage for each whole integer increment of material advantage. The material advantage of each position was determined using the following piece values: P = 1, N = B = 3, R = 5, Q = 9. Only moves after the 24th ply were considered (to partially avoid influence from opening books), and each material advantage had to exist for at least 6 ply. The winning percentage for when the material was even was set to 50%, based on the assumption that the difference of the winning percentage in such a case (54.7%) is entirely due to White advantage. For subsequent material advantages, the winning percentages for White and for Black were used in the following formula to produce a centered (my term) winning percentage: winning percentage = W% - avg(W% - B%) + 50%. This formula is used to mitigate the influence of White advantage. For example, the win percentage when White was ahead 2 pawns worth of material was 79.8%; for Black, the percentage was 25.3%. The resulting percentage used was 79.8% - ((79.8% + 25.3%)/2) +50% = 77.3%.
*Notes:
1) Material advantage can be thought of as the same as pawn advantage, since the value of a pawn is equal to the basic unit of material advantage.
2) The winning percentages are actually White's winning percentage. Thus, 25.3% was White's score when Black had a 2 pawn advantage.
3) 100% - winning percentage was used as the win percentage for negative material advantages. Thus, when the material advantage was -2 pawns, the winning percentage used was (100% - 77.3%) = 22.8%.
I plotted the resulting data and compared it to the following logistic model: Win Percentage = 1 / ( 1 + 10 ^(Pawn advantage/4))
The adjusted R squared value for this fit is 0.9983 and the root mean square error is 0.0156.
That model is a very good fit for the data. And when we compare this model to the model used to predict the expected score of a match based on Elo difference ( Expected score = 1 / ( 1 + 10 ^ (Elo Difference/400)) ), we can interpret each point of Elo difference to be analogous to a centiPawn. One should keep in mind that this is an average over all games where an X centipawn material advantage persisted for 6 plies. For a specific position, the true value of a paticular material advantage depends on the number and arrangement of the pieces on the board.
I should report that many of these games were adjudicated early (winner declared by GUI when score is - X centipawns for Y consecutive moves). Presumably, this would have no different effect than if the losing engine resigned. In reality, given that not all engines use table bases and might make a mistake (thus causing a won game to be drawn), there probably is some effect.
One thing that I have not studied or given thought to yet is exactly why my data, using the relative piece values I gave above, and Pradu's and Sune's data, using P=1, N=4. B=4.1, R=6, Q=12, should agree so well. I will think about this, and welcome anybody else to give a reason.
I collected over 900,000 long time control engine vs engine games. Then, using Norm Pollock's utilities, I filtered the games so that each opponent was greater than 2700 Elo (CCRL scale) and, for each engine pairing, the two opponents were within 50 Elo of each other. This produced a database of 343,789 games to study.
Using ChessDB (a branch of Scid by Dr. David Kirby), I determined the winning percentage for each whole integer increment of material advantage. The material advantage of each position was determined using the following piece values: P = 1, N = B = 3, R = 5, Q = 9. Only moves after the 24th ply were considered (to partially avoid influence from opening books), and each material advantage had to exist for at least 6 ply. The winning percentage for when the material was even was set to 50%, based on the assumption that the difference of the winning percentage in such a case (54.7%) is entirely due to White advantage. For subsequent material advantages, the winning percentages for White and for Black were used in the following formula to produce a centered (my term) winning percentage: winning percentage = W% - avg(W% - B%) + 50%. This formula is used to mitigate the influence of White advantage. For example, the win percentage when White was ahead 2 pawns worth of material was 79.8%; for Black, the percentage was 25.3%. The resulting percentage used was 79.8% - ((79.8% + 25.3%)/2) +50% = 77.3%.
*Notes:
1) Material advantage can be thought of as the same as pawn advantage, since the value of a pawn is equal to the basic unit of material advantage.
2) The winning percentages are actually White's winning percentage. Thus, 25.3% was White's score when Black had a 2 pawn advantage.
3) 100% - winning percentage was used as the win percentage for negative material advantages. Thus, when the material advantage was -2 pawns, the winning percentage used was (100% - 77.3%) = 22.8%.
I plotted the resulting data and compared it to the following logistic model: Win Percentage = 1 / ( 1 + 10 ^(Pawn advantage/4))
The adjusted R squared value for this fit is 0.9983 and the root mean square error is 0.0156.
That model is a very good fit for the data. And when we compare this model to the model used to predict the expected score of a match based on Elo difference ( Expected score = 1 / ( 1 + 10 ^ (Elo Difference/400)) ), we can interpret each point of Elo difference to be analogous to a centiPawn. One should keep in mind that this is an average over all games where an X centipawn material advantage persisted for 6 plies. For a specific position, the true value of a paticular material advantage depends on the number and arrangement of the pieces on the board.
I should report that many of these games were adjudicated early (winner declared by GUI when score is - X centipawns for Y consecutive moves). Presumably, this would have no different effect than if the losing engine resigned. In reality, given that not all engines use table bases and might make a mistake (thus causing a won game to be drawn), there probably is some effect.
One thing that I have not studied or given thought to yet is exactly why my data, using the relative piece values I gave above, and Pradu's and Sune's data, using P=1, N=4. B=4.1, R=6, Q=12, should agree so well. I will think about this, and welcome anybody else to give a reason.