Determine errors that would have been encountered if required confidence for sample size was met (when it wasn't)

[SerenityNetworks]

In my case, I need to perform an accuracy audit where my required accuracy is 99.90% and my required confidence level is 90.00%. I'm stumped on how to assign penalties in a case where the measured accuracy passes the requirement, but the required confidence fails.

I know that if I have a sample size of 3889, and I only have 1 error, then the formula "=1-BINOM.DIST(1,3889,1-0.999,TRUE)" will tell me my confidence level is 90.01%. My measured accuracy would be "=1-(1/3889)" or 99.97%. This meets both my measured accuracy and confidence level requirements.

But if I have 2 errors then my confidence level is 75.53%, even though the measured accuracy is 99.95%. So my accuracy is okay, but it's not at the required confidence level.

My issue is that I need to assign penalties for failing to meet the required accuracy and confidence, but the penalties are based only on the number of errors beyond the required accuracy. With a sample set of 3889 I have to get 4 errors before my measured accuracy drops below 99.90%, even though the required confidence fails with both 2 and 3 errors. I'm stumped on how to assign penalties in a case where the measured accuracy passes the requirement, but the required confidence fails.

Is it possible, and would it be correct, to be able to calculate backwards to determine 'n' by saying something like, "With 3889 samples at 74.53% confidence, then it's expected that 'n' errors would have occurred if the confidence were at 90.00%." If so, how would this be done? Would there be a margin of error that could be calculated? If so, how? I'm stumped on how to move forward with this problem.

I'd appreciate any experienced thoughts on this matter.
Andrew

 

[SerenityNetworks]

Never mind. Solved.

 

[jamiemarie]

Could you please explain how this is solved in case someone else with a similar problem comes across this thread?

 

[SerenityNetworks]

Could you please explain how this is solved in case someone else with a similar problem comes across this thread?

Yes. I should have the time in the next couple of days.

 

[SerenityNetworks]

The audit being performed is a detection audit. For each sample, there is either a detection or there is not. If there is not then this constitutes and error.

To pass the audit, both the required accuracy and required confidence must be met. There are penalties when the audit fails. The problem is that the penalties are calculated based upon only the measured accuracy. Thus when the required accuracy is met, but the required accuracy is not, then the calculated penalty is zero - even though the audit fails.

After investigation, there does not seem to be a statistical method for calculating a penalty when the required accuracy is met, but the required confidence is not. This seems to be more of a management decision than a mathematical method. One possible solution is to calculate the missing penalties by using a ratio. (See Columns U and V in the screenshot below.)