I did a spreadsheet for all simplifications on that site. I then set June 1st ELO to zero. I then summed the ELO gain ( or loss) for all simplification from June First to December First. The net ELO loss was 24.54 ELO over a six month period. IMHO this is too much ELO loss. Six months is too much time to wait for a statistical correction. The easiest way to fix this is to slightly alter the simplification bounds on the the tests. E.g. changing the bounds from [-3.00, 1.00] to something like [-2.50, 1.50] would favor less ELO loss (or a shorter time for a correction to occur). It might make simplification slightly harder to pass but it reduces the chances of having long runs of simplifications that have a net ELO loss.Uri Blass wrote: ↑Thu Dec 12, 2019 7:09 pmThis is about the linkZenmastur wrote: ↑Thu Dec 12, 2019 1:42 pmI have no clue what all this is supposed to mean.Uri Blass wrote: ↑Thu Dec 12, 2019 3:56 am
There are not enough games to know if a simplification is a regression or an improvement but you can get an unbiased estimate for the average value of simplifications from stockfish10.
These are the first numbers and you need to get more numbers from the link and calculate average for that purpose.
At least when I look at the first numbers it seems to me that the average is positive.
209.73->207.78(-1.95 elo) 1.12.2018 simplification
208.88->206.03(-2.85 elo) 6.12.2018 simplification
208.75->211.58(2.83 elo) 16.12.2018 simplification
214.03->216.00(1.97 elo) 16.12.2018 simplification
214.25->213.08(-1.17 elo) 24.12.2018 simplification
212.66->213.98(1.32 elo) 27.12.2018 simplification
209.88->210.54(0.66 elo) 4.1.2019 simplification
211.45->215.12(3.67 elo) 10.1.2019 simplification
215.12->212.84(-2.28 elo) 14.1.2019 simplification
212.84->212.17(-0.67 elo) 14.1.2019 simplification
212.17->216.75(4.58 elo) 17.1.2019 simplification
215.25->217.07(1.82 elo) 22.1.2019 simplification
216.10->215.10(-1 elo) 29.1.2019 simplification
215.10->221.39(6.29 elo) 31.1.2019 simplification
217.75->219.64(1.89 elo) 8.2.2019 simplification
219.64->220.48(0.84 elo) 21.2.2019 simplification
220.48->220.45(-0.03 elo) 21.2.2019 simplification
218.64->219.64(1 elo) 27.2.2019 simplification
220.93->218.38->220.45(-0.48 elo) 5.3 simplifications
219.49->220.93(+1.44 elo) 10.3 simplification
219.87->218.20(-1.67 elo) 20.3 simplification
221.09->218.53(-2.56 elo) 24.3 simplification
217.85->218.81(0.96 elo) 4.4 simplification
223.36->220.86->221.82(-1.54 elo) 13.4 simplifications
219.64->219.14->218.61->219.15(-0.49 elo) 16.4 smplifications
219.15->220.30(1.15 elo) 17.4 simplification
218.51->218.61(0.1 elo) 19.4 simplification
221.37->220.81->225.70(4.33 elo) 9.5 simplifications
I will explain one line and you can understand the other lines based on the same logic
209.73->207.78(-1.95 elo) 1.12.2018 simplification
The following lines are from the link
https://nextchessmove.com/dev-builds
20181201-0929 20000 11146 433 8421 +207.78 +/- 3.64 Simplification
20181129-1517 20000 11271 478 8251 +209.73 +/- 3.69 Non Functional
209.73 is elo difference from stockfish7 before the simplification.
207.78 is elo difference from stockfish7 after the simplification.
-1.95 is the estimate for elo improvement from the simplification(note that the statistical mistake is above 3.6 elo).
1.12.2018 is the date of the simplification.
The idea is that you can get unbiased estimate for the elo that stockfish get from simplifications by the sum of all these numbers.
I did not calculate the sum of all these numbers but at least the sum of the numbers that I wrote that is only about part of the simplications is above 0.
Maybe somebody can continue to calculate the sum of all the numbers(there are many simplifications after the simplification of 9.5.2019 when I did not write the numbers but you can get it from the link).
Regards,
Zenmastur