Yes, it should have been 100 although, as you noted, the end result doesn't change. This is my first program in Python and I was writing it in one window while I was googling for python tutorials/documentation in another Luckily Python is very easy to learn and comes very handy for such things. For me, used to C/C++, it is like to park the truck and take the bicycle, really nice.
I would like to ask interested people that keeps big log files of the test match between 2 engines (at least 10K games per test) to run them through this script and report the results, together with testing conditions. So to verify if good testing framework are distinguishable from bad ones just looking at the results of the script.
Noise in ELO estimators: a quantitative approach
Moderators: hgm, Rebel, chrisw
-
- Posts: 2684
- Joined: Sat Jun 14, 2008 9:17 pm
-
- Posts: 937
- Joined: Fri Mar 10, 2006 4:29 pm
- Location: Germany
Re: Noise in ELO estimators: a quantitative approach.
Hi Marco,
you know I'm running gauntlets against different opponents when testing different versions of SF. Does your tool work in such cases, too?
Here is the output of your Noise_Estimator for one of these testruns:
This was the test for SF-edce2a8. TC 15"+0.1"
Best, Jörg.
you know I'm running gauntlets against different opponents when testing different versions of SF. Does your tool work in such cases, too?
Here is the output of your Noise_Estimator for one of these testruns:
Code: Select all
Games: 2796 , result: [957, 902, 937]
Estimated ELO: -0.0
Noise as function of number of games:
[924.79, 285.34, 54.43, 48.36, 127.66, 200.24, 42.3, 66.6, 90.65, 30.2, 48.32, 57.41, 39.25, 54.39, 45.34, 54.39, 42.28, 42.28, 21.13, 27.18, 27.18]
Best, Jörg.
Jörg Oster
-
- Posts: 2684
- Joined: Sat Jun 14, 2008 9:17 pm
Re: Noise in ELO estimators: a quantitative approach.
Hi Jörg, unfortunatly at the moment it expects only two engines tests.
Here is some output from mines, always 15" per game, single thread.
Here is some output from mines, always 15" per game, single thread.
Code: Select all
Games: 2377 , result: [456, 475, 1446]
Estimated ELO: -6.94963842777
Noise as function of number of games:
[11.35, 5.67, 16.8, 2.65, 3.58, 3.96, 3.96, 3.77, 0.57, 0.57, 0.19]
-
- Posts: 568
- Joined: Tue Dec 12, 2006 10:10 am
- Full name: Gary Linscott
Re: Noise in ELO estimators: a quantitative approach.
Interesting, my noise numbers are quite a bit higher with concurrency 3. This is on a 4 core machine, all engines running single threaded. Here are the results from two runs:mcostalba wrote:Hi Jörg, unfortunatly at the moment it expects only two engines tests.
Here is some output from mines, always 15" per game, single thread.
Code: Select all
Games: 2377 , result: [456, 475, 1446] Estimated ELO: -6.94963842777 Noise as function of number of games: [11.35, 5.67, 16.8, 2.65, 3.58, 3.96, 3.96, 3.77, 0.57, 0.57, 0.19]
Code: Select all
Games: 1699 , result: [262, 342, 1095]
Estimated ELO: -16.6897337053
Noise as function of number of games:
['217.72', '61.94', '42.47', '27.59', '29.63', '6.72', '57.93', '49.01', '56.17', '41.48', '15.70', '28.53', '33.47']
Code: Select all
Games: 1751 , result: [300, 345, 1106]
Estimated ELO: -9.03536155887
Noise as function of number of games:
['173.24', '47.80', '44.15', '57.67', '47.29', '30.85', '29.70', '18.46', '28.18', '7.44', '11.36', '25.56', '12.30']
-
- Posts: 568
- Joined: Tue Dec 12, 2006 10:10 am
- Full name: Gary Linscott
Re: Noise in ELO estimators: a quantitative approach.
And here is a 6000 game result:
Code: Select all
Games: 6063 , result: [945, 1219, 3899]
Estimated ELO: -15.9933239309
Noise as function of number of games:
['152.80', '195.34', '156.53', '42.58', '15.93', '93.26', '29.86', '23.20', '53.41', '16.39', '5.89', '9.33', '13.94', '6.22', '4.19', '19.04', '35.93', '44.54', '54.09', '82.65', '34.68', '30.58', '12.08', '13.87', '33.28', '33.59', '40.82', '13.74', '13.99', '12.54', '4.76', '3.88', '4.27', '2.60', '4.39', '5.55', '9.39', '14.05', '16.62', '20.75', '10.56', '7.69', '10.08', '8.17', '5.84', '3.06', '2.09']
-
- Posts: 2684
- Joined: Sat Jun 14, 2008 9:17 pm
Re: Noise in ELO estimators: a quantitative approach.
They seem a bit slow to converge.gladius wrote:And here is a 6000 game result:
Would be interesting to see the noise figure in case you keep same conditions but without concurrency.