# How to Verify a Problem Is Related to a Specific Test Environment?

Six Sigma – iSixSigma Forums General Forums Tools & Templates How to Verify a Problem Is Related to a Specific Test Environment?

Viewing 4 posts - 1 through 4 (of 4 total)
• Author
Posts
• #245678

ChrisHoll
Participant

We have a software test environment that is experiencing an intermittent failure of 5 in 40 trials (1/8).  We suspect this is not a software problem, but something in the environment.  If we move the software to a new environment (or release it), how many tests need to be run to ensure (to, say 95%) that the problem is not in the software?  I’ve tried looking up various confidence/reliability tools, but none seem to address this question.  (Or I don’t know how to apply this problem to those tools.)  If we run 40 trials and see no failures that gives a certain comfort level, but I’d like to quantify that, and/or run fewer trials.

0
#245682

Robert Butler
Participant

The question you are asking is one of binary proportions. Since you have a sample probability for failure (p = r/n where r = number of failures and n = total number of trials) in the one environment you also have a measure of the standard deviation of that proportion  = sqrt([p*(1-p)]/n).  You also have a measure of the proportion of failures in a different environment where you took 40 samples and found 0 failures.  You can use this information to compare the two proportions as well as determine the number of samples needed to make statements concerning the odds that the failure rate in the new environment is zero and the degree of certainty that the failure rate is 0.

Rather than try to provide an adequate summary of the mathematics needed to do this I would recommend you look up the subject of proportion comparison and sample size calculations for estimates of differences in proportions.  I can’t point to anything on the web but I can recommend two books that cover the subject – you should be able to get both of these through inter-library loan.

1. An Introduction to Medical Statistics – 3rd Edition – Bland

2. Statistical Methods for Rates and Proportions – 3rd Edition – Fleiss, Levin, Paik

0
#245683

ChrisHoll
Participant

Thank you Robert.  This gives me a place to start.  So p = 12.5% and std = .0523.  We haven’t run any tests in the other environments yet.  The question is how many need to be run to provide 95% confidence that the problem was related to the first environment.   5 tests?  10?  I’ll try to learn more.

0
#245796

Chris Butterworth
Participant

Have a look at the binomial distribution for n = 40 and p = 0.125. You can see that it is reasonable to observe 5 defects in a sample of 40.

But observing zero defects is improbable – it occurs less than 0.5% of the time.