SF17 still holding its own

[nmcrazyim5]

With so much Elo gain across many engines recently, I wanted to see how Stockfish with the older, smaller net from a couple of years ago would fare against them. Surprisingly, Stockfish 17 is still holding its own against other top engines (Torch and Leela excluded) in my testing against the very latest builds of Reckless, PlentyChess, and Integral.

Conditions
TC: 1m+1s
Book: UHO 2024
Threads: 8
Hash: 1024MB
CPU: AMD Ryzen 9 7950X

Cross-table from 100-pair gauntlet

Code: Select all

Rank Name                          Elo     +/-   Games   Score    Draw 
   0 Stockfish 17                   29      23     600   54.2%   33.3% 
   1 Reckless 0.10.0 dev             0      40     200   50.0%   30.0% 
   2 PlentyChess 8.0               -24      39     200   46.5%   35.0% 
   3 Integral 8                    -63      39     200   41.0%   35.0% 

As you can see, Reckless dev and SF17 were tied after 100 pairs.
However, Stefan has the Reckless version from April at +38 celos stronger than SF17: https://www.sp-cc.de/files/uho_full_list.txt
(The version I used is the latest commit from a couple of weeks ago, which on paper should be stronger than the April version in the SPCC list.)

As I understand it, SF17 and Reckless260401 never met during testing because there were later SF releases by then, and SF17's Celo was calculated when it was released. I'm curious to know if this is a feature of the way the Celo gets calculated (SF17 played relatively weaker opponents in 2024, pushing its Celo down) or if there is notable SSS in my testing and we should expect Reckless to come out ahead by +38 elos given enough games (seems unlikely)?

[nmcrazyim5]

@pohl4711 would love to hear your thoughts on this.