sprt question

[elcabesa]

hi,
I need some help with SPRT. To test my engine I'm using openchess by Andy Grant. I'd like to ply a regression test and openbench uses sprt to stop the games. I don't want to exit from the framework, but I don't wnat that SPRT will stop my game too early,

I'm now trying this test: a game with e0 = 0 , e1 = 0, alpha and beta that give SPRT Bounds of (-11.51, 11.51).
there are some better methods to achiee this?

[Ratosh]

Hi,

I've a modification on his OpenBench on my fork that i can input the minimum number of games i want to use in a test, i use it to make regression tests with a fixed number of games.

This is my modifications: Change

PS.: You'll need to patch your database to create the mingames column.

[Ajedrecista]

Hello Marco:

hi,
I need some help with SPRT. To test my engine I'm using openchess by Andy Grant. I'd like to ply a regression test and openbench uses sprt to stop the games. I don't want to exit from the framework, but I don't wnat that SPRT will stop my game too early,

I'm now trying this test: a game with e0 = 0 , e1 = 0, alpha and beta that give SPRT Bounds of (-11.51, 11.51).
there are some better methods to achiee this?

I am not a SPRT expert but I thought that e0 < e1 is compulsory and the expected length of the test is somewhat proportional to 1/(e1 - e0)². Furthermore, if you get symmetric bounds (-11.51 and 11.51), you are using alpha = beta, and in this special case bounds = ±ln|alpha/(1 - alpha)|, so alpha = 1/[1 + exp(-|bound|)] = beta. In your case: alpha = beta = 1/[1+ exp(-11.51)] ~ 1e-6 (0.0001% of false positives and 0.0001% of false negatives), which is ridicously low and test will go forever in combination with e0 = e1, which I thought it would not work.

My advice is modify the settings, SF framework is a good reference point. Alpha = 0.05 = beta (bounds ~ ± 2.94, like SF) and e1 - e0 greater than SF settings (because I guess that you have less resources than them) look reasonable. I would add that you should avoid e1 = -e0 because SPRT is a coin flip then and you lose advantage of SPRT, not said by me but by someone else that knew more than me in SF Google group when it started few years ago.

If you want to test patches that are supposed to be improvements, e0 + e1 > 0 is the way to go, while simplifications could be tested with e0 + e1 < 0 if you do not mind to lose Elo in average at the benefit of simplifying/reduce code that will be easier to maintain.

Regards from Spain.

Ajedrecista.

[AndrewGrant]

Hi,

I've a modification on his OpenBench on my fork that i can input the minimum number of games i want to use in a test, i use it to make regression tests with a fixed number of games.

This is my modifications: Change

PS.: You'll need to patch your database to create the mingames column.

+1, I would do the same if I had the need and was not lazy