Page 2 of 2
Re: Noise in ELO estimators: a quantitative approach.
Posted: Fri Jan 18, 2013 9:01 pm
by mcostalba
Yes, it should have been 100 although, as you noted, the end result doesn't change. This is my first program in Python and I was writing it in one window while I was googling for python tutorials/documentation in another
Luckily Python is very easy to learn and comes very handy for such things. For me, used to C/C++, it is like to park the truck and take the bicycle, really nice.
I would like to ask interested people that keeps big log files of the test match between 2 engines (at least 10K games per test) to run them through this script and report the results, together with testing conditions. So to verify if good testing framework are distinguishable from bad ones just looking at the results of the script.
Re: Noise in ELO estimators: a quantitative approach.
Posted: Fri Jan 18, 2013 10:52 pm
by Joerg Oster
Hi Marco,
you know I'm running gauntlets against different opponents when testing different versions of SF. Does your tool work in such cases, too?
Here is the output of your Noise_Estimator for one of these testruns:
Code: Select all
Games: 2796 , result: [957, 902, 937]
Estimated ELO: -0.0
Noise as function of number of games:
[924.79, 285.34, 54.43, 48.36, 127.66, 200.24, 42.3, 66.6, 90.65, 30.2, 48.32, 57.41, 39.25, 54.39, 45.34, 54.39, 42.28, 42.28, 21.13, 27.18, 27.18]
This was the test for SF-edce2a8. TC 15"+0.1"
Best, Jörg.
Re: Noise in ELO estimators: a quantitative approach.
Posted: Sat Jan 19, 2013 9:30 am
by mcostalba
Hi Jörg, unfortunatly at the moment it expects only two engines tests.
Here is some output from mines, always 15" per game, single thread.
Code: Select all
Games: 2377 , result: [456, 475, 1446]
Estimated ELO: -6.94963842777
Noise as function of number of games:
[11.35, 5.67, 16.8, 2.65, 3.58, 3.96, 3.96, 3.77, 0.57, 0.57, 0.19]
Re: Noise in ELO estimators: a quantitative approach.
Posted: Sun Jan 20, 2013 2:44 am
by gladius
mcostalba wrote:Hi Jörg, unfortunatly at the moment it expects only two engines tests.
Here is some output from mines, always 15" per game, single thread.
Code: Select all
Games: 2377 , result: [456, 475, 1446]
Estimated ELO: -6.94963842777
Noise as function of number of games:
[11.35, 5.67, 16.8, 2.65, 3.58, 3.96, 3.96, 3.77, 0.57, 0.57, 0.19]
Interesting, my noise numbers are quite a bit higher with concurrency 3. This is on a 4 core machine, all engines running single threaded. Here are the results from two runs:
Code: Select all
Games: 1699 , result: [262, 342, 1095]
Estimated ELO: -16.6897337053
Noise as function of number of games:
['217.72', '61.94', '42.47', '27.59', '29.63', '6.72', '57.93', '49.01', '56.17', '41.48', '15.70', '28.53', '33.47']
Code: Select all
Games: 1751 , result: [300, 345, 1106]
Estimated ELO: -9.03536155887
Noise as function of number of games:
['173.24', '47.80', '44.15', '57.67', '47.29', '30.85', '29.70', '18.46', '28.18', '7.44', '11.36', '25.56', '12.30']
Re: Noise in ELO estimators: a quantitative approach.
Posted: Mon Jan 21, 2013 3:42 pm
by gladius
And here is a 6000 game result:
Code: Select all
Games: 6063 , result: [945, 1219, 3899]
Estimated ELO: -15.9933239309
Noise as function of number of games:
['152.80', '195.34', '156.53', '42.58', '15.93', '93.26', '29.86', '23.20', '53.41', '16.39', '5.89', '9.33', '13.94', '6.22', '4.19', '19.04', '35.93', '44.54', '54.09', '82.65', '34.68', '30.58', '12.08', '13.87', '33.28', '33.59', '40.82', '13.74', '13.99', '12.54', '4.76', '3.88', '4.27', '2.60', '4.39', '5.55', '9.39', '14.05', '16.62', '20.75', '10.56', '7.69', '10.08', '8.17', '5.84', '3.06', '2.09']
Re: Noise in ELO estimators: a quantitative approach.
Posted: Mon Jan 21, 2013 7:38 pm
by mcostalba
gladius wrote:And here is a 6000 game result:
They seem a bit slow to converge.
Would be interesting to see the noise figure in case you keep same conditions but without concurrency.