# How to Verify a Problem Is Related to a Specific Test Environment?

Six Sigma – iSixSigma › Forums › General Forums › Tools & Templates › How to Verify a Problem Is Related to a Specific Test Environment?

- This topic has 3 replies, 3 voices, and was last updated 2 months, 1 week ago by Chris Butterworth.

- AuthorPosts
- January 22, 2020 at 5:21 pm #245678

ChrisHollParticipant@ChrisHoll**Include @ChrisHoll in your post and this person will**

be notified via email.We have a software test environment that is experiencing an intermittent failure of 5 in 40 trials (1/8). We suspect this is not a software problem, but something in the environment. If we move the software to a new environment (or release it), how many tests need to be run to ensure (to, say 95%) that the problem is not in the software? I’ve tried looking up various confidence/reliability tools, but none seem to address this question. (Or I don’t know how to apply this problem to those tools.) If we run 40 trials and see no failures that gives a certain comfort level, but I’d like to quantify that, and/or run fewer trials.

0January 22, 2020 at 6:36 pm #245682

Robert ButlerParticipant@rbutler**Include @rbutler in your post and this person will**

be notified via email.The question you are asking is one of binary proportions. Since you have a sample probability for failure (p = r/n where r = number of failures and n = total number of trials) in the one environment you also have a measure of the standard deviation of that proportion = sqrt([p*(1-p)]/n). You also have a measure of the proportion of failures in a different environment where you took 40 samples and found 0 failures. You can use this information to compare the two proportions as well as determine the number of samples needed to make statements concerning the odds that the failure rate in the new environment is zero and the degree of certainty that the failure rate is 0.

Rather than try to provide an adequate summary of the mathematics needed to do this I would recommend you look up the subject of proportion comparison and sample size calculations for estimates of differences in proportions. I can’t point to anything on the web but I can recommend two books that cover the subject – you should be able to get both of these through inter-library loan.

1. An Introduction to Medical Statistics – 3rd Edition – Bland

2. Statistical Methods for Rates and Proportions – 3rd Edition – Fleiss, Levin, Paik

0January 22, 2020 at 6:56 pm #245683

ChrisHollParticipant@ChrisHoll**Include @ChrisHoll in your post and this person will**

be notified via email.Thank you Robert. This gives me a place to start. So p = 12.5% and std = .0523. We haven’t run any tests in the other environments yet. The question is how many need to be run to provide 95% confidence that the problem was related to the first environment. 5 tests? 10? I’ll try to learn more.

0January 27, 2020 at 11:54 am #245796

Chris ButterworthParticipant@belfieldlad**Include @belfieldlad in your post and this person will**

be notified via email.Have a look at the binomial distribution for n = 40 and p = 0.125. You can see that it is reasonable to observe 5 defects in a sample of 40.

But observing zero defects is improbable – it occurs less than 0.5% of the time.

I hope this is helpful.

Chris Butterworth

0 - AuthorPosts

You must be logged in to reply to this topic.