The question of computing passing probabilities for SPRT tests often comes up and people then resort to simulation. Alternatively one could use my script
that computes passing probabilities and average running times. It has no external dependencies. For instructions on how to use sprta.py I include a copy of the post I made on the SF forum.
Here is a simple plain python script for computing passing probabilities and average running times for SPRT tests
Michel wrote:The question of computing passing probabilities for SPRT tests often comes up and people then resort to simulation. Alternatively one could use my script
that computes passing probabilities and average running times. It has no external dependencies. For instructions on how to use sprta.py I include a copy of the post I made on the SF forum.
Here is a simple plain python script for computing passing probabilities and average running times for SPRT tests
Can you make the script also output the LOS value for results of the matches at SPRT stop? The asymptotic value of LOS is (1 + erf[{wins - losses}/sqrt{2*(wins+losses)}] ) / 2, and is O(1) computation. The exact LOS is a sum over binomials, a O(N^2) computation, but asymptotically (N->infinity) they are the same. The reason I would like to see LOS is that I observed the typical LOS values for alpha, beta = 0.05, usually the SPRT stop occurs at LOS of 0.990-0.995. Although Type I error is theoretically unbounded for LOS or p-value as stopping rule, practically, say in the range N=100 games to 100,000 games, Type I error can be kept to 2-5% for LOS=0.999 or such.
Laskos wrote:
Can you make the script also output the LOS value for results of the matches at SPRT stop? The asymptotic value of LOS is (1 + erf[{wins - losses}/sqrt{2*(wins+losses)}] ) / 2, and is O(1) computation. The exact LOS is a sum over binomials, a O(N^2) computation, but asymptotically (N->infinity) they are the same. The reason I would like to see LOS is that I observed the typical LOS values for alpha, beta = 0.05, usually the SPRT stop occurs at LOS of 0.990-0.995. Although Type I error is theoretically unbounded for LOS or p-value as stopping rule, practically, say in the range N=100 games to 100,000 games, Type I error can be kept to 2-5% for LOS=0.999 or such.
You need to understand the hypothesis that lie behind the formula you're using. The "LOS" (actually p-value) is biaised at the SPRT stopping point. The only way your formula is correct, is when you run a fixed number of games decided in advance: (X1...Xn) need to be n random variable, independant and identically distributed, where n is not random (and especially not a function of the X(t) process).
Of course, the biais converges to zero as the stopping time goes to infinity. But still, it's biaised, especially when stopping happens early. That's why we don't print the p-value of an SPRT test.
Theory and practice sometimes clash. And when that happens, theory loses. Every single time.
Laskos wrote:
Can you make the script also output the LOS value for results of the matches at SPRT stop? The asymptotic value of LOS is (1 + erf[{wins - losses}/sqrt{2*(wins+losses)}] ) / 2, and is O(1) computation. The exact LOS is a sum over binomials, a O(N^2) computation, but asymptotically (N->infinity) they are the same. The reason I would like to see LOS is that I observed the typical LOS values for alpha, beta = 0.05, usually the SPRT stop occurs at LOS of 0.990-0.995. Although Type I error is theoretically unbounded for LOS or p-value as stopping rule, practically, say in the range N=100 games to 100,000 games, Type I error can be kept to 2-5% for LOS=0.999 or such.
You need to understand the hypothesis that lie behind the formula you're using. The "LOS" (actually p-value) is biaised at the SPRT stopping point. The only way your formula is correct, is when you run a fixed number of games decided in advance: (X1...Xn) need to be n random variable, independant and identically distributed, where n is not random (and especially not a function of the X(t) process).
Of course, the biais converges to zero as the stopping time goes to infinity. But still, it's biaised, especially when stopping happens early. That's why we don't print the p-value of an SPRT test.
If LOS stop rule is "I stop as soon as I get LOS 0.999 for the first time", then the bias is included in the Type I error derived by simulations. As for p-value given by above formula, it converges to LOS for large number of games for fixed number of games.
Laskos wrote:
Can you make the script also output the LOS value for results of the matches at SPRT stop? The asymptotic value of LOS is (1 + erf[{wins - losses}/sqrt{2*(wins+losses)}] ) / 2, and is O(1) computation. The exact LOS is a sum over binomials, a O(N^2) computation, but asymptotically (N->infinity) they are the same. The reason I would like to see LOS is that I observed the typical LOS values for alpha, beta = 0.05, usually the SPRT stop occurs at LOS of 0.990-0.995. Although Type I error is theoretically unbounded for LOS or p-value as stopping rule, practically, say in the range N=100 games to 100,000 games, Type I error can be kept to 2-5% for LOS=0.999 or such.
You need to understand the hypothesis that lie behind the formula you're using. The "LOS" (actually p-value) is biaised at the SPRT stopping point. The only way your formula is correct, is when you run a fixed number of games decided in advance: (X1...Xn) need to be n random variable, independant and identically distributed, where n is not random (and especially not a function of the X(t) process).
Of course, the biais converges to zero as the stopping time goes to infinity. But still, it's biaised, especially when stopping happens early. That's why we don't print the p-value of an SPRT test.
If LOS stop rule is "I stop as soon as I get LOS 0.999 for the first time", then the bias is included in the Type I error derived by simulations. As for p-value given by above formula, it converges to LOS for large number of games for fixed number of games.
By the way, your statement that frequentist inference LOS is actually p-value is not correct. Just computed the p-value for our problem, and it's only related to frequentist inference LOS.
