Noise in ELO estimators: a quantitative approach

Discussion of chess software programming and technical issues.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
mcostalba
Posts: 2684
Joined: Sat Jun 14, 2008 7:17 pm

Re: Noise in ELO estimators: a quantitative approach.

Post by mcostalba » Fri Jan 18, 2013 8:01 pm

Yes, it should have been 100 although, as you noted, the end result doesn't change. This is my first program in Python and I was writing it in one window while I was googling for python tutorials/documentation in another :-) Luckily Python is very easy to learn and comes very handy for such things. For me, used to C/C++, it is like to park the truck and take the bicycle, really nice.

I would like to ask interested people that keeps big log files of the test match between 2 engines (at least 10K games per test) to run them through this script and report the results, together with testing conditions. So to verify if good testing framework are distinguishable from bad ones just looking at the results of the script.

Joerg Oster
Posts: 684
Joined: Fri Mar 10, 2006 3:29 pm
Location: Germany

Re: Noise in ELO estimators: a quantitative approach.

Post by Joerg Oster » Fri Jan 18, 2013 9:52 pm

Hi Marco,

you know I'm running gauntlets against different opponents when testing different versions of SF. Does your tool work in such cases, too?
Here is the output of your Noise_Estimator for one of these testruns:

Code: Select all

Games: 2796 , result: [957, 902, 937]
Estimated ELO: -0.0
Noise as function of number of games:
[924.79, 285.34, 54.43, 48.36, 127.66, 200.24, 42.3, 66.6, 90.65, 30.2, 48.32, 57.41, 39.25, 54.39, 45.34, 54.39, 42.28, 42.28, 21.13, 27.18, 27.18]
This was the test for SF-edce2a8. TC 15"+0.1"

Best, Jörg.
Jörg Oster

mcostalba
Posts: 2684
Joined: Sat Jun 14, 2008 7:17 pm

Re: Noise in ELO estimators: a quantitative approach.

Post by mcostalba » Sat Jan 19, 2013 8:30 am

Hi Jörg, unfortunatly at the moment it expects only two engines tests.

Here is some output from mines, always 15" per game, single thread.

Code: Select all

Games: 2377 , result: [456, 475, 1446]
Estimated ELO: -6.94963842777
Noise as function of number of games:
[11.35, 5.67, 16.8, 2.65, 3.58, 3.96, 3.96, 3.77, 0.57, 0.57, 0.19]

gladius
Posts: 538
Joined: Tue Dec 12, 2006 9:10 am

Re: Noise in ELO estimators: a quantitative approach.

Post by gladius » Sun Jan 20, 2013 1:44 am

mcostalba wrote:Hi Jörg, unfortunatly at the moment it expects only two engines tests.

Here is some output from mines, always 15" per game, single thread.

Code: Select all

Games: 2377 , result: [456, 475, 1446]
Estimated ELO: -6.94963842777
Noise as function of number of games:
[11.35, 5.67, 16.8, 2.65, 3.58, 3.96, 3.96, 3.77, 0.57, 0.57, 0.19]
Interesting, my noise numbers are quite a bit higher with concurrency 3. This is on a 4 core machine, all engines running single threaded. Here are the results from two runs:

Code: Select all

Games: 1699 , result: [262, 342, 1095]
Estimated ELO: -16.6897337053
Noise as function of number of games:
['217.72', '61.94', '42.47', '27.59', '29.63', '6.72', '57.93', '49.01', '56.17', '41.48', '15.70', '28.53', '33.47']

Code: Select all

Games: 1751 , result: [300, 345, 1106]
Estimated ELO: -9.03536155887
Noise as function of number of games:
['173.24', '47.80', '44.15', '57.67', '47.29', '30.85', '29.70', '18.46', '28.18', '7.44', '11.36', '25.56', '12.30']

gladius
Posts: 538
Joined: Tue Dec 12, 2006 9:10 am

Re: Noise in ELO estimators: a quantitative approach.

Post by gladius » Mon Jan 21, 2013 2:42 pm

And here is a 6000 game result:

Code: Select all

Games: 6063 , result: [945, 1219, 3899]
Estimated ELO: -15.9933239309
Noise as function of number of games:
['152.80', '195.34', '156.53', '42.58', '15.93', '93.26', '29.86', '23.20', '53.41', '16.39', '5.89', '9.33', '13.94', '6.22', '4.19', '19.04', '35.93', '44.54', '54.09', '82.65', '34.68', '30.58', '12.08', '13.87', '33.28', '33.59', '40.82', '13.74', '13.99', '12.54', '4.76', '3.88', '4.27', '2.60', '4.39', '5.55', '9.39', '14.05', '16.62', '20.75', '10.56', '7.69', '10.08', '8.17', '5.84', '3.06', '2.09']

mcostalba
Posts: 2684
Joined: Sat Jun 14, 2008 7:17 pm

Re: Noise in ELO estimators: a quantitative approach.

Post by mcostalba » Mon Jan 21, 2013 6:38 pm

gladius wrote:And here is a 6000 game result:
They seem a bit slow to converge.

Would be interesting to see the noise figure in case you keep same conditions but without concurrency.

Post Reply