TalkChess.com

Alessandro Scotti wrote:I seem to have reached a plateau with Hamsters where new features and even bug fixes hardly contribute any elo to it.
This would bother me little if not for the fact that each and every test starts by feeding me the illusion of improvement, and only later falls to the same old stats.
A 700 games test tournament might go like this:
- games 0-100: 53% (ehi, not bad)
- games 101-200: 54% (great!)
- games 201-300: 54% (yo-hoo!... goes out to buy champagne!)
- games 301-400: 53% (just a little glitch!)
- games 401-500: 52% (some bad luck here...)
- games 501-600: 51% (you son of a...)
- games 601-700: 50% (nooooooooooooooo!!!)
So it's just my imagination or this happen to you too?!? One would think that 400 games already provide a good approximation, yet...

Alessandro Scotti wrote:I seem to have reached a plateau with Hamsters where new features and even bug fixes hardly contribute any elo to it.
This would bother me little if not for the fact that each and every test starts by feeding me the illusion of improvement, and only later falls to the same old stats.
A 700 games test tournament might go like this:
- games 0-100: 53% (ehi, not bad)
- games 101-200: 54% (great!)
- games 201-300: 54% (yo-hoo!... goes out to buy champagne!)
- games 301-400: 53% (just a little glitch!)
- games 401-500: 52% (some bad luck here...)
- games 501-600: 51% (you son of a...)
- games 601-700: 50% (nooooooooooooooo!!!)
So it's just my imagination or this happen to you too?!? One would think that 400 games already provide a good approximation, yet...

Alessandro Scotti wrote:I seem to have reached a plateau with Hamsters where new features and even bug fixes hardly contribute any elo to it.
This would bother me little if not for the fact that each and every test starts by feeding me the illusion of improvement, and only later falls to the same old stats.
A 700 games test tournament might go like this:
- games 0-100: 53% (ehi, not bad)
- games 101-200: 54% (great!)
- games 201-300: 54% (yo-hoo!... goes out to buy champagne!)
- games 301-400: 53% (just a little glitch!)
- games 401-500: 52% (some bad luck here...)
- games 501-600: 51% (you son of a...)
- games 601-700: 50% (nooooooooooooooo!!!)
So it's just my imagination or this happen to you too?!? One would think that 400 games already provide a good approximation, yet...

hgm wrote:Did you ever solve this puzzle, Ed?

Not satisfactory. I reran the whole thing at a higher time control (40/20) and the problem disappeared. But doing so 800 games lasted 800 hours which was unacceptable for me, even if you have 4 PC's at your disposal at the time.

My educated guess as that due to tiny Windows interferences stealing 100-200ms so now and then (the infamous windows swap-file comes to mind) programs go one iteration deeper or 1 ply less deeper than a former game resulting in different moves, thus randomness plays a role after all.

The higher time control you play, the fewer such things occurs.


It seems that +/- 1% is really a bit better than you can expect over 800 games: the standard error in a single run should be 0.4/sqrt(800) = 1.4%, but the difference of the result of two independent runs should be sqrt(2) larger, i.e. 2%. And in 32% of the cases the difference would be larger than that.

If I calculate the spread of the 3 percentages you quote, it is 1.9%. This is a bit larger than expected, but not so much that it is wildly improbable.

But one might argue that the different runs are not sufficiently independent. If this is the case one would really have to know the experimental spread of the results of the run under exactly the same condition.

If it turns out to be a real effect, the only thing I can think of is that the OS is to blame. I don't know under what conditions you played these matches (ponder on/off, single/dual core), but if there is competition between the engines for CPU and/or memory, the OS might systematically favor one because of its position in the process table (determined by the order of starting them). Or the GUI might of course cheat, giving a higher priority to the engine it starts with...