Zenmastur wrote:But I'm still curious about using an asymmetrical CI. If it's not too complicated would it be possible for you to try to optimize the efficiency while allowing each tail to assume a value independent of the other tail? E.G. for a 95% CI both tales are usually 0.025 wide instead I'm thinking that with a highly skewed acceptance ratio, like 10%, accepting 1% of bad patches while rejecting 4% of good patches would produce greater efficiency. I'm sure 1% and 4% aren't the right numbers but it would be nice to know how much more efficient an optimized pair would be.
Regards,
Zen
You are up to something, Zen, I found that Type II error (false negatives,
fn here) can be much laxer than Type I error (false positives,
fp here), in proportion similar to your guess. The result could spare Stockfish framework of 20% workload if the proportion of good patches (
Pgood here) is above 10%-15%. I estimate that proportion of good patches is about 20% in SF tests during last several weeks.
Similarly to symmetric CI, the efficiency is proportional to:
Code: Select all
(fp (-1 + Pgood) + Pgood - fn Pgood)/(-Log[fn/(1 - fp)] + Log[(1 - fn)/fp])
For Pgood=20%, the shape of efficiency as a function of Type I error and Type II error is:
It's already visible that the maximum efficiency is somewhere for small Type I error, but pretty large Type II error.
To pinpoint the optima, I took the first derivative on both
fp and
fn, and luckily it wasn't necessary to compute second derivatives and the Hessian.
Code: Select all
ContourPlot[{(fp + (-1 + fn - fp) Pgood + (-1 + fp) fp (-1 + Pgood) (-Log[fn/(1 - fp)] + Log[(1 - fn)/fp]))/((-1 + fp) fp (Log[fn/(1 - fp)] - Log[(1 - fn)/fp])^2) == 0, (fp + (-1 + fn - fp) Pgood + (-1 + fn) fn Pgood (Log[fn/(1 - fp)] - Log[(1 - fn)/fp]))/((-1 + fn) fn (Log[fn/(1 - fp)] - Log[(1 - fn)/fp])^2) == 0}]
The 3D contour plot is useless, one cannot see anything clearly there, so I just plotted 2D contour plots for several reasonable values of
Pgood (proportion of good patches), the maximum efficiency is achieved at the intersection of the 2 curves. We see that Type II error can safely be taken as 15% or higher if the proportion of good patches is above 10%, while Type I error is similarly strict to symmetrical CI (much smaller than Type II error).
The intersection is the optimum for efficiency, clickable thumbnails.
Pgood=10%
Pgood=15%
Pgood=20%
Pgood=25%
Pgood=30%
Pgood=40%
Absurd case Pgood=51% (just submit them all instantly), no solution