Any application that involves searching where testing changes for improvement is a major fraction of the work done AND the acceptance rate of changes is highly skewed would benefit. Obviously other games that use minmax searches would be a prime example. I'm sure that there would be other applications as well. I don't know anything about LaTeX, but I suppose this isn't the only application that can be used to write a paper. Even if the only place the paper found a home was on the chess programming Wiki it would be worth writing. It would probably need an extensive introduction section that details the testing framework as well as the mechanics of implementation. Anyway it was just an idea.Laskos wrote: If you think this thing is worth something, and are not averse to typing in LaTeX (I am), then that's good, your contribution is anyway crucial in posing the problem and having the idea that Type II error might behave differently. I did this for fun and hastily, in fact I would need to re-check everything.
I actually thought about this exact graph yesterday before going to bed but I was too tired to post anything else. Again... nice job!
If Pgood drops below 10% using the current bounds it looks as if efficiency could easily drop an order of magnitude compared to the optimal interval if I'm reading the graph correctly.
If you could predict/calculate the probability of a patch being accepted or rejected then the CI could be adjusted on a per patch basis. This could lead to huge reductions in the number of test games required and a huge increase in the ELO gain per test game played. So, it's probably worth looking into how this might be accomplished.
I have several other ideas but its almost 5:00am ... well past my bedtime and I'm not likely to be thinking very clearly.. so maybe after I get some sleep.
Regards,
Zen