Re: Engine Testing - Statistics
Posted: Mon Jan 18, 2010 12:57 am
Yes. If your bankroll, or Elo goes positive, you can stop the test and conclude you are better, but that is not valid. Play enough hands, or enough games, and your bankroll goes to zero, or the Elo drops to below where it was previously.BubbaTough wrote:I understand the fallacy you are referring to, and that many gamblers fall into. I am just not sure the analogy is fitting just right.bob wrote: The idea is this. If you start a session, and play until you are one unit ahead, and then quit, you are one unit up. And with the significant variance in the game of 21, the odds of your going up at one particular instant are quite high. And each time you do this, you win one bet. Suppose you play on a $5 min bet table. Every session will end up at +5 bucks. Except for the occasional session where you start off with a loss and continue to play and lose.
The point is that lifetime winnings occurs over a lifetime, counting everything. If you quit because you are ahead, you start off at the same probability of winning the next hand whether you quit and don't play until tomorrow, or you stay at the table and play the next hand. But I know of many gamblers that swear they are winning overall, just by playing the game without any sort of "advantage play" to really give them an actual edge over the house. It is always the "I win more sessions than I lose" idea. If you play 20 sessions, of 500 hands each (3-4-5 hours of play depending on crowd) you play 10,000 hands. If you flat-bet $10 per hand, you then had $100,000 of "action". and the house will keep pretty close to $500 bucks of your money. It doesn't matter whether you change that and play 1 session of 10,000 hands, or 10,000 sessions of one hand each. You end up the same $500 in the hole.
Now if you do as some con-artists do, and play multiple sessions and report the number of times you walked away a winner vs a lower, you can win more sessions, while still losing that same 500 bucks, because you can lose more in one session than you win in all the rest. But if you claim "more winning sessions than losing sessoins" a "noob" will buy that book, which is the way the author gets rich, not by playing the game with his losing system. And the "noob" will verify that he wins more sessions than he loses, all the time asking himself "where in the devil is my money going?"
If you test and when you like the result, you stop, all you did was find a place where the god of variance was kind to you, and reach the conclusion you wanted to reach. If you play toward some set goal (say N thousand games, then you ignore that variance, and the more games you play, the smaller that variance becomes.
If you look at the data I posted the other day, after a thousand games, I could have stopped, and been "sure" that this version was better, because the variance had taken the Elo up quite a bit. It did come down to reality, given enough games as you could see.
Is the "bankroll" equivalent to ELO?
That's the idea. The "game" never ends. You just break it up into chunks. A single small chunk can show most anything, while if you play long enough, you break the casino or they break you...Is the "stopping after you are down $500" equivalent to stopping your test when you are down a certain number of games?
No. In blackjack, we often refer to "n0" which is the number of games required to get you to approach the expected value with a +1.0% edge. Typical values are 15-20,000 hands to give you small enough SD to make -1SD a still positive bankroll effect...
Is the "lifetime" equivalent to playing an infinity number of games in your test?
No, I actually have watched the error bar, and decided how many games are needed to make the error bar small enough to be useful. For huge rating differences, you don't need a small error bar to reach a 95% confidence that A is worse than B or vice-versa.. But for more reasonable values, say 10-20 Elo at the most, a few hundred just won't do.
I guess its a little silly for me to nitpick the analogy. I think what is throwing me is that, generally speaking in gambling, you figure out your odds with math, and deduce how control your bankroll based on that. If I am understanding the analogy correctly, in computer chess you are kind of doing the opposite. You are watching your results, and trying to deduce the math.
-Sam