SPRT questions

Uri Blass · Post by **Uri Blass** » Fri May 15, 2015 7:06 am

one result from the stockfish framework

LLR: 0.72 (-2.94,2.94) [0.00,4.50]
Total: 233707 W: 38339 L: 37483 D: 157885

1)Is there a simple calculator to check if this result is enough to pass
[0.00 b] for some smaller bound than 4.50?

2)What happens if people make a test that simply stop if you pass SPRT
with bounds [0,b] for some positive bound for b<=6 and fail if you fail SPRT with bounds [0,b] for some positive bound for b<=6.

Can people calculate what is the theoretical probability for a regression of 1 elo to pass the test and what is the probability of 0 elo to pass the test?

Same also for the expected number of games and what is the worst case.

People claim that SPRT is the best but it is not clear for what
and in the example of Lucas's patch it is not clear to me if it is not simply better to accept the patch of Lucas and stop the test.

What is the important additional information that we get if the test pass or fail.

We may not know that the patch is positive with 95% confidence but I think that we can know that there is even not 0.5 elo regression with 95% confidence for every possible result and at the same time it is possible that we have 1.5 elo improvement for every possible result.

3)Did somebody test if the worst case really behave in the way that SPRT expects?

A possible way is to test the program against itself(so we know 0 elo is the right result) with SPRT(-b,b) many times to see if the distribution of the length of games is really the distribution that theory expect.

I suspect that the worse case is practically worse at least if both programs play the same opening with white and black every time
and I see no reason not to do it because not doing it increase the variety of the result and it is better to reduce the variety of the result(when of course using SPRT is not the correct way to continue because the assumption of independent results is not correct)

Zenmastur · Post by **Zenmastur** » Fri May 15, 2015 10:59 am

Kai Laskos is working on these problems. See the thread labeled "
Maximum ELO gain per test game played?" I would link it but I don't know how.

Apparently the model set-up is more tedious than he anticipated, but I believe he has the equations in hand.

See his last 4 or 5 posts.

Regards,

Zen

Ajedrecista · Post by **Ajedrecista** » Fri May 15, 2015 1:02 pm

Hello Uri:

I will try to answer some of your questions in a random order:

------------------------

Uri Blass wrote:Can people calculate what is the theoretical probability for a regression of 1 elo to pass the test and what is the probability of 0 elo to pass the test?

Same also for the expected number of games and what is the worst case.

Yes, it is possible thanks to Michel's Python script sprta.py (he also made a C file so you can compile it and build an executable). Please take a look here for more details. It returns the probability of pass and the expected number of games.

The worst case (the largest expected number of games) is when Bayeselo = (Bayeselo_0 + Bayeselo_1)/2 in a SPRT(Bayeselo_0, Bayeselo_1) case (Bayeselo_0 < Bayeselo_1 always). Michel's script has an special output: Elo gain [logistic Elo, i.e. Elo] and SPRT bounds [Bayeselo].

------------------------

Uri Blass wrote:one result from the stockfish framework

LLR: 0.72 (-2.94,2.94) [0.00,4.50]
Total: 233707 W: 38339 L: 37483 D: 157885

1)Is there a simple calculator to check if this result is enough to pass
[0.00 b] for some smaller bound than 4.50?

I wrote a LLR calculator long time ago. But you must keep in mind that SPRT is sequential so the history of the games counts (it is not the same WWWLLL than WLWLWL). The answer to your question is yes, it exists such calculator.

In SF testing framework: alpha = 0.05 = beta in SPRT. So only changing Bayeselo_1: