sprt question

Discussion of chess software programming and technical issues.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
Post Reply
elcabesa
Posts: 827
Joined: Sun May 23, 2010 11:32 am
Contact:

sprt question

Post by elcabesa » Sat Dec 15, 2018 12:17 pm

hi,
I need some help with SPRT. To test my engine I'm using openchess by Andy Grant. I'd like to ply a regression test and openbench uses sprt to stop the games. I don't want to exit from the framework, but I don't wnat that SPRT will stop my game too early,

I'm now trying this test: a game with e0 = 0 , e1 = 0, alpha and beta that give SPRT Bounds of (-11.51, 11.51).
there are some better methods to achiee this?

Ratosh
Posts: 72
Joined: Mon Apr 16, 2018 4:56 pm

Re: sprt question

Post by Ratosh » Sat Dec 15, 2018 1:13 pm

Hi,

I've a modification on his OpenBench on my fork that i can input the minimum number of games i want to use in a test, i use it to make regression tests with a fixed number of games.

This is my modifications: Change

PS.: You'll need to patch your database to create the mingames column.

User avatar
Ajedrecista
Posts: 1405
Joined: Wed Jul 13, 2011 7:04 pm
Location: Madrid, Spain.
Contact:

Re: SPRT question.

Post by Ajedrecista » Sun Dec 16, 2018 11:09 am

Hello Marco:
elcabesa wrote:
Sat Dec 15, 2018 12:17 pm
hi,
I need some help with SPRT. To test my engine I'm using openchess by Andy Grant. I'd like to ply a regression test and openbench uses sprt to stop the games. I don't want to exit from the framework, but I don't wnat that SPRT will stop my game too early,

I'm now trying this test: a game with e0 = 0 , e1 = 0, alpha and beta that give SPRT Bounds of (-11.51, 11.51).
there are some better methods to achiee this?
I am not a SPRT expert but I thought that e0 < e1 is compulsory and the expected length of the test is somewhat proportional to 1/(e1 - e0)². Furthermore, if you get symmetric bounds (-11.51 and 11.51), you are using alpha = beta, and in this special case bounds = ±ln|alpha/(1 - alpha)|, so alpha = 1/[1 + exp(-|bound|)] = beta. In your case: alpha = beta = 1/[1+ exp(-11.51)] ~ 1e-6 (0.0001% of false positives and 0.0001% of false negatives), which is ridicously low and test will go forever in combination with e0 = e1, which I thought it would not work.

My advice is modify the settings, SF framework is a good reference point. Alpha = 0.05 = beta (bounds ~ ± 2.94, like SF) and e1 - e0 greater than SF settings (because I guess that you have less resources than them) look reasonable. I would add that you should avoid e1 = -e0 because SPRT is a coin flip then and you lose advantage of SPRT, not said by me but by someone else that knew more than me in SF Google group when it started few years ago.

If you want to test patches that are supposed to be improvements, e0 + e1 > 0 is the way to go, while simplifications could be tested with e0 + e1 < 0 if you do not mind to lose Elo in average at the benefit of simplifying/reduce code that will be easier to maintain.

Regards from Spain.

Ajedrecista.

AndrewGrant
Posts: 497
Joined: Tue Apr 19, 2016 4:08 am
Location: U.S.A
Full name: Andrew Grant
Contact:

Re: sprt question

Post by AndrewGrant » Mon Dec 17, 2018 4:01 am

Ratosh wrote:
Sat Dec 15, 2018 1:13 pm
Hi,

I've a modification on his OpenBench on my fork that i can input the minimum number of games i want to use in a test, i use it to make regression tests with a fixed number of games.

This is my modifications: Change

PS.: You'll need to patch your database to create the mingames column.
+1, I would do the same if I had the need and was not lazy

Post Reply