Anti-cheating

[Guenther]

I think the interesting idea is to train a neural network to detect engine/human games. After you train it on a large enough dataset, it will be relatively easy to check statistics for false positives for such a network.

Well, that's exactly what they did for lichess.
https://github.com/clarkerubber/irwin

[Cornfed]

This is "friendly discussion" rather than "expert help", but isn't detection of cheating mainly down to absence of shallow tactical errors?

Certainly not, anti-cheating requires a proper statistical model.

Absolutely correct.

And, 'proper' is more complicated than many a chess player thinks.

I know on chess.com (and less on lichess where I have not played in a while), I used Aimchess to give stats on my last 1000+ blitz (3 min/no delay) games and I am almost always 'better' than my peers on all counts - except TIME MANAGMENT. I loose far more games than I should on time...ENDGAMES as well and that is often because I am blundering away better/equal endings...due to being low on time I am sure.

[scchess]

I think the interesting idea is to train a neural network to detect engine/human games. After you train it on a large enough dataset, it will be relatively easy to check statistics for false positives for such a network.

Well, that's exactly what they did for lichess.
https://github.com/clarkerubber/irwin

Correct. Lichess, true to the spirit, did publish the source code open. However, it's useless to the outside world in practice. There is no documentation the input features and what exactly the model is predicting. More importantly, there is no mention at the data format.

[scchess]

This is "friendly discussion" rather than "expert help", but isn't detection of cheating mainly down to absence of shallow tactical errors?

Certainly not, anti-cheating requires a proper statistical model.

Absolutely correct.

And, 'proper' is more complicated than many a chess player thinks.

I know on chess.com (and less on lichess where I have not played in a while), I used Aimchess to give stats on my last 1000+ blitz (3 min/no delay) games and I am almost always 'better' than my peers on all counts - except TIME MANAGMENT. I loose far more games than I should on time...ENDGAMES as well and that is often because I am blundering away better/equal endings...due to being low on time I am sure.

Exactly. Anti-cheating, in my imagination, is a discipline of data mining. The system would mine everything possible from a public dataset such as Lichess's dataset, and private dataset which may include labelled data. The system would then compare your own performance statistically against everybody else to work out if you are a statistical outlier.

A simple toy-example is "z-score". Given a null hypothesis that you're not a cheater, how does your performance stand against everybody else in your rating group on a scaled normal distribution? Are we confident that you're a cheater because you're performance is 5 standard deviations away? How does the false positive rates look like if the rule is 5 SD, how does it look like if it's 3 SD? Can we plot something like a ROC curve? What'd the confusion matrix look like?

[scchess]

Another issue is how to estimate the complexity of a position. This is a very important feature for anti-cheating but it's hard. Chess.com definitely has a way to estimate the complexity. I remember there was an old paper using the frequency of PV switching during iterative deepening as an estimation. Also, the depth of the search and width of the search could also be used.

The Maia paper published last year has some insights. I quote:

... One way to measure this complexity is to consider the difference in quality between the best and second best moves in the position. If this difference is small there are at least two good moves to choose from, and if it’s large there is only
one good move. ...

The ability not to blunder, the ability to follow computer PV lines as the complexity of the position vary with time pressure is going to be a key factor in anti-cheating. There should be a correlation, we should be able to fit a curve like how Stockfish fits its WDL sigmoid curve. Anyone going outside this curve statistically could be a cheater.

[scchess]

I think the interesting idea is to train a neural network to detect engine/human games. After you train it on a large enough dataset, it will be relatively easy to check statistics for false positives for such a network.

Well, that's exactly what they did for lichess.
https://github.com/clarkerubber/irwin

Correct. Lichess, true to the spirit, did publish the source code open. However, it's useless to the outside world in practice. There is no documentation the input features and what exactly the model is predicting. More importantly, there is no mention at the data format.

While I don't have lichess's data. It ---- looks ---- like the model is predicting if someone is a cheater or not, a binary decision. For example, https://github.com/clarkerubber/irwin/b ... ng.py#L105 looks like forming an array of binary labels. The loss function is binary_crossentropy (https://github.com/clarkerubber/irwin/b ... del.py#L81) so the assumption looks correct.

By far the most important in the project is the input data. It looks like the project requires a connection to lichess' labelled cheater in their DB (https://github.com/clarkerubber/irwin/b ... ing.py#L42). The input data seems to be analyzed PGN data from Fishnet (https://github.com/clarkerubber/irwin/b ... me.py#L163).

For the actual model, it looks like an LSTM model. https://github.com/clarkerubber/irwin/b ... meModel.py. The model looks alright if the input data is game moves, because the game moves indeed form a sequence.

Overall, it looks like the project takes game analysis data from Fishnet to form a binary classification. Doesn't look like it takes the player's performance history, but I could be wrong. There's definitely more cheating code somewhere else in the main part of the lichess project.

[jdart]

Since most online computer players use Stockfish, a correlation between game moves and Stockfish-selected moves would also be interesting.

Of course, a lot of opening nowadays follows Stockfish recommendations, but a human player following known theory for a long series of moves is also a tell: most players do not have that level of opening knowledge.

[Cornfed]

Exactly. Anti-cheating, in my imagination, is a discipline of data mining.

I remember in my online poker days a program called Poker Tracker - data mining really. It more than paid for itself - so many times over.

I might add concerning online chess, time is recorded and used in various ways as well to detect cheaters. I would hazard to guess that close to 99% of all people deemed 'cheaters' online, really and truly are. For everyone though, sites (I believe) do let you get another membership under a different name. That's surely a better deal than our legal system.

[scchess]

Since most online computer players use Stockfish, a correlation between game moves and Stockfish-selected moves would also be interesting.

Of course, a lot of opening nowadays follows Stockfish recommendations, but a human player following known theory for a long series of moves is also a tell: most players do not have that level of opening knowledge.

Yes, but the positional complexity will need to be factored in. GM-level players can see someone cheats if the player can play out tactically perfect moves in highly complicated positions that even themself can't evaluate correctly. That's exactly why a simple accuracy check like "94% matched to Stockfish!" is useful but insufficient.

[scchess]

Exactly. Anti-cheating, in my imagination, is a discipline of data mining.

I remember in my online poker days a program called Poker Tracker - data mining really. It more than paid for itself - so many times over.

I might add concerning online chess, time is recorded and used in various ways as well to detect cheaters. I would hazard to guess that close to 99% of all people deemed 'cheaters' online, really and truly are. For everyone though, sites (I believe) do let you get another membership under a different name. That's surely a better deal than our legal system.

Exactly! It's ok to leave out false negatives, we will never able to catch everybody. However, false positive rates must be controlled statistically. If a system believes there is a cheating occurred, then it must be able to do it very confidently.