What can be said about 1 - 0 score?
Posted: Tue Mar 28, 2017 8:07 am
It happens to me that I have only one game between 2 engines. It gets even worse, engines have an unknown difference in strength, and I generally don't know anything about them. No theoretical or empiric prior. But I have the moves in the game. Here I use only the number of moves until one engine mates another in that single game (no adjudication). What I can say about the general relative strength of the engines?
1/ Expectation value of the score.
Expectation value for win: (1+1)/(3+1) = 1/2
Expectation value for draw: (0+1)/(3+1) = 1/4
Expectation value for loss: (0+1)/(3+1) = 1/4
Expectation value for Score: 1/2+1/8 = 5/8
Likelihood of Success (LOS): (1!0!/1!+1/2*1!0!/1!)/2 = 3/4
2/ Maximum Likelihood Estimation of the ELO difference by number of moves.
Here I plot likelihoods for the number of moves to the mate in Stockfish-Zurichess games (1200 Elo points difference) and Stockfish-Stockfish games (0 Elo points difference):
We see that if the game ended in mate in 40 moves, the likelihood that one engine is stronger by 1200 Elo points is an order of magnitude larger than the likelihood that the engines are equal. The number of moves at maximum likelihood for 1200 Elo points difference is 32, for 0 Elo points difference is 77 moves. And the linear interpolation is:
So, if this single game we have ended in mate in 40 moves, the maximum likelihood estimation would be that engines are separated by something like 1000 Elo points. I think everybody justly feels that it is a very rough estimation, only useful when we don't know anything about the engines.
1/ Expectation value of the score.
Expectation value for win: (1+1)/(3+1) = 1/2
Expectation value for draw: (0+1)/(3+1) = 1/4
Expectation value for loss: (0+1)/(3+1) = 1/4
Expectation value for Score: 1/2+1/8 = 5/8
Likelihood of Success (LOS): (1!0!/1!+1/2*1!0!/1!)/2 = 3/4
2/ Maximum Likelihood Estimation of the ELO difference by number of moves.
Here I plot likelihoods for the number of moves to the mate in Stockfish-Zurichess games (1200 Elo points difference) and Stockfish-Stockfish games (0 Elo points difference):
We see that if the game ended in mate in 40 moves, the likelihood that one engine is stronger by 1200 Elo points is an order of magnitude larger than the likelihood that the engines are equal. The number of moves at maximum likelihood for 1200 Elo points difference is 32, for 0 Elo points difference is 77 moves. And the linear interpolation is:
So, if this single game we have ended in mate in 40 moves, the maximum likelihood estimation would be that engines are separated by something like 1000 Elo points. I think everybody justly feels that it is a very rough estimation, only useful when we don't know anything about the engines.