Determine a players rating by the moves he makes...

[JBNielsen]

Can you determine a players rating by the moves he makes...?

Todays PCs/Chessprograms are so strong, that their deep analysis can almost be considered as the truth.
If you compare a players moves (the computers evaluation of the moves) with the evaluation of what the computer considers the best move, there will be some difference.

And I assume, the higher rating the player has, the lower will the average difference be.

Has anyone tried to find the relation between the average difference and a players rating?

Of course it is best to use as many games/moves by the player as possible.

Perhaps this estimate of a rating is already incorporated in some programs?

I have just bought Chess Assistant with Rybka4, but it has not anything like that.

[Laskos]

Can you determine a players rating by the moves he makes...?

Todays PCs/Chessprograms are so strong, that their deep analysis can almost be considered as the truth.
If you compare a players moves (the computers evaluation of the moves) with the evaluation of what the computer considers the best move, there will be some difference.

And I assume, the higher rating the player has, the lower will the average difference be.

Has anyone tried to find the relation between the average difference and a players rating?

Of course it is best to use as many games/moves by the player as possible.

Perhaps this estimate of a rating is already incorporated in some programs?

I have just bought Chess Assistant with Rybka4, but it has not anything like that.

Probably one could separate a 2700 player from 1800 one, but not much more. I tried to do some correlation tests, the GMs playing more computerish moves were not necessarily stronger than GMs playing less computer-like, I didn't observe any correlation at this level. Not only the computers themselves are playing differently, but the game of chess has a different flow and meaning to strong humans compared to comps. Planning is one feature, and there are many which determine un-computer-like moves of many strong GMs or even generally humans.

Kai

[Christopher Conkie]

Can you determine a players rating by the moves he makes...?

Todays PCs/Chessprograms are so strong, that their deep analysis can almost be considered as the truth.
If you compare a players moves (the computers evaluation of the moves) with the evaluation of what the computer considers the best move, there will be some difference.

And I assume, the higher rating the player has, the lower will the average difference be.

Has anyone tried to find the relation between the average difference and a players rating?

Of course it is best to use as many games/moves by the player as possible.

Perhaps this estimate of a rating is already incorporated in some programs?

I have just bought Chess Assistant with Rybka4, but it has not anything like that.

Probably one could separate a 2700 player from 1800 one, but not much more. I tried to do some correlation tests, the GMs playing more computerish moves were not necessarily stronger than GMs playing less computer-like, I didn't observe any correlation at this level. Not only the computers themselves are playing differently, but the game of chess has a different flow and meaning to strong humans compared to comps. Planning is one feature, and there are many which determine un-computer-like moves of many strong GMs or even generally humans.

Kai

You would agree then that such a thing cannot be done in software. I have known a few times in my life where the Moon came into conjunction with Neptune and I could almost believe I was better than I was.

I know one thing. That if you are into computer chess you do (subliminally) start to play like a computer (however badly).

Chris

[JBNielsen]

How many games did you look at?

I think a minimum would be 10 games for each of 5 players with ratings of 2000, 2200, 2400, 2600 and 2800.

Sometimes you play 200 above your rating; sometimes 200 below.
So a certain amount of games is needed, and the results cannot be used to determine a players rating based on a single game.

- - -

It could also be interesting to see if a deep analysis (fx 14 ply?!) would indicate the improvement in Deep Blue in these 3 versions:

1) Copenhagen 1993 (in the 8 40m/2h games: 4 against Bent Larsen, and 4 against Lars Schandorf, Lars Bo Hansen, Carsten Høi and Henrik Danielsen)
2) 1996 - First match against Kasparov
3) 1997 - Second match against Kasparov

[Laskos]

How many games did you look at?

I think a minimum would be 10 games for each of 5 players with ratings of 2000, 2200, 2400, 2600 and 2800.

Sometimes you play 200 above your rating; sometimes 200 below.
So a certain amount of games is needed, and the results cannot be used to determine a players rating based on a single game.

- - -

It could also be interesting to see if a deep analysis (fx 14 ply?!) would indicate the improvement in Deep Blue in these 3 versions:

1) Copenhagen 1993 (in the 8 40m/2h games: 4 against Bent Larsen, and 4 against Lars Schandorf, Lars Bo Hansen, Carsten Høi and Henrik Danielsen)
2) 1996 - First match against Kasparov
3) 1997 - Second match against Kasparov

I used some ~40-50 games of several ~2550 Elo GMs compared to several ~2750 Elo GMs, the results were well within one standard deviation error of computer-like play (means no statistical difference at all). I guess a 1800 player would differ, but I never tried.

For Deep Blue, not the fixed depth, fixed time, say 10 minutes/move, would be meaningful, but all would be reduced to some sort of blunder-check, no more than that, how you determine the correct choice in unpreferential positions, with several PV lines differing by few centipawns, where even modern, converging engines differ in choice, not talking about Deep Blue. In fact, I bet the blunder check is the most useful thing we can do generally, not some statistical measures on computer-like play for determining the strength.

Kai

[Sarciness]

This has been attempted some time ago. It has its uses and limitations. The original site compared players' choices with a computer's choice at fixed depth and also accounted for complexity to some degree. I can't remember if it was linked to the chessmetrics site or not. Someone here will probably remember!

edit: this article will probably interest you: http://en.wikipedia.org/wiki/Methods_fo ... er_choices

[Mike S.]

Another large analysis project of that type was started - but AFAIK never finished - at the Chessninja message board.

http://www.chessninja.com/boards/

(Sorry, I don't have a direct URL to the related threads. But I am almost sure they are up, still.)

One important element of such analyses is, which moves to exclude. Because in some situations, (small) differences to engine evaluations are meaningless. For example the early opening choices, and positions which are very imbalanced already and masters may choose a safe & clear path to the win instead of a "study like" faster one.

Basically, I agree to the idea and I would even suppose that not very big depths are required. Big numbers of moves being evaluated should have priority.

[lkaufman]

This has been attempted some time ago. It has its uses and limitations. The original site compared players' choices with a computer's choice at fixed depth and also accounted for complexity to some degree. I can't remember if it was linked to the chessmetrics site or not. Someone here will probably remember!

edit: this article will probably interest you: http://en.wikipedia.org/wiki/Methods_fo ... er_choices

The biggest problem with this method of estimating ratings by the mean absolute or squared error according to a super-strong engine is that players like Tal who aim for wild positions will show much higher average errors than players like Karpov who aim for slight positional advantages. I do think that the method should work pretty well for rating the two opponents in a match, so you could take all the matches played for the world championship (and perhaps candidates' matches) and rate them sequentially based on mean error, since presumably if a game is quiet for White it is also quiet for Black. This should then permit comparison of ratings across time, perhaps starting at move 20 in each game to minimize the importance of memorized theory. The one slightly questionable assumption made with this method is that players in these matches do not get stronger or weaker between matches, which though certainly false in any specific instance maybe roughly true on average. There might be a bias towards modern players though in that the more modern player will necessarily always be younger than the earlier player when they meet in a match, and on average younger players improve relative to older players between matches.

[JBNielsen]

Thanks for the link, Ramdewar.

Especially this further link is very good:
http://web.zone.ee/chessanalysis/summary450.pdf

It shows a lot of interesting things; fx the absolute average errors by these players:

0.099 - 2700 player
0.139 - 2500 player
0.174 - 2300 player
0.186 - 2100 player

I can recommend reading this!

One important element of such analyses is, which moves to exclude. Because in some situations, (small) differences to engine evaluations are meaningless. For example the early opening choices, and positions which are very imbalanced already and masters may choose a safe & clear path to the win instead of a "study like" faster one.

Basically, I agree to the idea and I would even suppose that not very big depths are required. Big numbers of moves being evaluated should have priority.

The results in the above link has actually excluded the opening moves, and as it concerns +2100 players I assume, that only very few moves are played in obviously won positions because the loser would usually have resigned. So only a few safe moves should have been played instead of best moves.

So perhaps it is possible for an analyzing program to finish with something like "based on this analyzis your estimated performance-rating in this game is 2222".
It would be a nice feature, and perhaps it could motivate players(beginners) to improve their play.

[Laskos]

Thanks for the link, Ramdewar.

Especially this further link is very good:
http://web.zone.ee/chessanalysis/summary450.pdf

It shows a lot of interesting things; fx the absolute average errors by these players:

0.099 - 2700 player
0.139 - 2500 player
0.174 - 2300 player
0.186 - 2100 player

I can recommend reading this!

As I said, the blunder-check is more useful than statistical analysis of how computerish the moves are. Curious, though, what they consider to be an "error"? 10% errors in the games of 2700 players seem like a bit too much to be considered clear blunders. Another thing, if you observe carefully, the standard deviation is high. And still another thing, the player ratings according to this "error" check appear mangled, because of the different playing styles.

The final outcome is something like "according to this blunder-check your rating is 2100 +/- 300 points 95% confidence", something pretty useless. As I said, it's probably easy to separate with confidence using computers a 1800 player from a 2700 one, not much more.

Kai