Stockfish developmental progess on pause?

[Norm Pollock]

Stockfish developmental progress is tested at https://nextchessmove.com/dev-builds

Each change is tested against Stockfish 7 in a trial of 20,000 random games. The randomness leaves room for variation.

However, in the last 15 trials covering the last 15 changes, the Elo differential above Stockfish 7, has been under 365 elo points.

If we look at the 14 trials prior to the last 15 trials, 10 out of the 14 had an Elo differential above 365 points.

Something seems to have gone wrong after the versions of 2020.22.10. On that date, 2 trials were above 370, and since then none have been above 365. It could be due to randomness, or perhaps some bug was accidentally introduced.

[mwyoung]

Stockfish developmental progress is tested at https://nextchessmove.com/dev-builds

Each change is tested against Stockfish 7 in a trial of 20,000 random games. The randomness leaves room for variation.

However, in the last 10 trials covering the last 10 changes, the Elo differential above Stockfish 7, has been under 365 elo points.

If we look at the 12 trials prior to the last 10 trials, 9 out of the 12 had an Elo differential above 365 points.

Something seems to have gone wrong after the versions of 2020.22.10. On that date, 2 trials were above 370, and since then none have been above 365. It could be due to randomness or perhaps, or perhaps some bug was accidentally introduced.

Or maybe Stockfish has reached perfect play and no more progress is possible.

[Guenther]

Stockfish developmental progress is tested at https://nextchessmove.com/dev-builds

Each change is tested against Stockfish 7 in a trial of 20,000 random games. The randomness leaves room for variation.

However, in the last 15 trials covering the last 15 changes, the Elo differential above Stockfish 7, has been under 365 elo points.

If we look at the 14 trials prior to the last 15 trials, 10 out of the 14 had an Elo differential above 365 points.

Something seems to have gone wrong after the versions of 2020.22.10. On that date, 2 trials were above 370, and since then none have been above 365. It could be due to randomness, or perhaps some bug was accidentally introduced.

IMHO opinion SF7 is simply too weak meanwhile to get reliable results against a much stronger SF12 and newer dev versions.
The error bars will be larger and larger, if the target is >350 always. (also don't forget it is still just a kind of selfplay test)

They should update to at least SF9 or SF10 now at NCM.