Are my process results really any different than yours?
Six Sigma – iSixSigma › Forums › Industries › Healthcare › Are my process results really any different than yours?
 This topic has 5 replies, 3 voices, and was last updated 11 years, 1 month ago by Andrew Banks.

AuthorPosts

March 30, 2011 at 5:50 pm #53775
Andrew BanksParticipant@BBinNC Include @BBinNC in your post and this person will
be notified via email.Forum Members:
I have reached the limits of my statistical knowledge and need some help. The question in the subject line is often asked, and in this case may even be appropriate. A team I completed a DMAIC project with has vastly improved their process (8fold reduction in defect rate & one tailed 2 proportions test of before/after data yields p<0.000) and held the results for 7 months now, but I have been asked by the process owner if the new process performance is the "same" as the rest of the hospital. This is not a metric the project planned to address, so I gave no thought as to how I would do this analysis upfront...
The variable in question is attribute, has 2 possible outcomes (good/bad), one trial per patient, but certainly I could envision scenarios in which the trials may not be truly independent or have the same probability for success.
So, I was able to get the last 6 months of data for this process for the entire hospital in summarized form (events/trials).
Now, how to test my hypothesis? Should I:
1. consider the entire hospital the population and use a 1proportion test (is it really the population)
2. subtract my teams department from the entire hospital data and then run the 2proprtions test (the sample sizes differ by a factor of 10 and Im not sure I can reasonably get the raw data to take random samples)
3. or what about using an ANOM technique where I use a pchart to determine if either category is different than the overall using 3 standard deviations?I have tried all 3, and the results lead to different conclusions (well, option 3 does anyway). So, which is the most appropriate interpretation of the situation (and therefore which test is most valid)? I have been tempted to ask, why does the answer to this question matter anyway?, but I know that at least in part the process owner is attempting to prioritize this metric against others for next projects, which is important to me.
Thanks for your help / input.
0April 6, 2011 at 4:57 pm #191424
Robert ButlerParticipant@rbutler Include @rbutler in your post and this person will
be notified via email.I’d recommend going with #2 and, if your software will produce them, generate the 95% confidence intervals for your proportion so that you can show where the proportion for the rest of the hospital falls relative to those intervals. If it is available, you could run the same thing but with a comparison against national standards.
I had to do something very similar to what you are describing about two weeks ago and by showing the location of the rest of the hospital as well as the location of the national rates relative to the confidence intervals of the proportions we had generated I was able to show that, in spite of our very small samples (the issue was that of a rare occurrence of an event), the data did indicate a clinically significant improvement. I also ran a posthoc power test on the proportions. The power wasn’t 80% but it was in the mid 60’s and, given what we had to work with, it was/is very encouraging.
0April 6, 2011 at 6:29 pm #191425
Andrew BanksParticipant@BBinNC Include @BBinNC in your post and this person will
be notified via email.Robert:
thanks for the input: I have followed your advice, and while the result is not what the process owner had hoped for, I can adequately describe the gap that still exists.
Regards,
0April 10, 2011 at 3:49 am #191434
SeverinoParticipant@Jsev607 Include @Jsev607 in your post and this person will
be notified via email.Robert Butler wrote:
I’d recommend going with #2 and, if your software will produce them, generate the 95% confidence intervals for your proportion so that you can show where the proportion for the rest of the hospital falls relative to those intervals. If it is available, you could run the same thing but with a comparison against national standards.
I had to do something very similar to what you are describing about two weeks ago and by showing the location of the rest of the hospital as well as the location of the national rates relative to the confidence intervals of the proportions we had generated I was able to show that, in spite of our very small samples (the issue was that of a rare occurrence of an event), the data did indicate a clinically significant improvement. I also ran a posthoc power test on the proportions. The power wasn’t 80% but it was in the mid 60’s and, given what we had to work with, it was/is very encouraging.
Really? I would’ve said #3. Isn’t the purpose of an ANOM to compare the average of different groups to the overall average and determine if a significant difference exists? Isn’t this exactly what he was asked to do or am I just stupid?
0April 12, 2011 at 1:39 pm #191440
Robert ButlerParticipant@rbutler Include @rbutler in your post and this person will
be notified via email.Yes, the object of ANOM is to compare various group means/proportions to their overall mean/proportion but the request, as stated in the first post was to see, “if the new process performance is the “same” as the rest of the hospital.”
My interpretation of this request is that the desired comparisons is that of two proportions (the hospital proportion less the group of interest and the particular group proportion) relative to one another and not that of taking the entire hospital, computing an overall proportion, and then running an analysis of all of various hospital groups, including the group of interest, against a grand proportion.
0April 12, 2011 at 3:53 pm #191441
Andrew BanksParticipant@BBinNC Include @BBinNC in your post and this person will
be notified via email.Robert:
I’m a little confuddled (yes, that’s a word of my own creation). I think I’ve reached a conclusion, but I could be way off. Any help getting out of the mire is greatly appreciated…
I know that a 2proportions test and the ANOM technique are fundamentally different (testing the independence of two binomial variables versus comparing them both to the overall average), but does it make any sense that they should “roughly” agree (i.e. analogous to the 2sample T and ANOVA) when the number of proportions being compared is reduced to 2 (and the test difference,H0, is 0)? Well, that was the thought I went with anyway.
My software (Minitab) offers 2 tests for 2proportions (one based on normal approx. and Fisher’s Exact Test). The ANOM I created by summarizing the data (defects,trials) in each “category”, (36,2724) and (174,22682) and then created the pchart.
I stated in my first post that the ANOM led me to a different conclusion. That was partly in error – I had not set the “control limits” at the alpha=0.05 level (roughly 2 “standard deviations”) to match the 2proportions test. When I made this adjustment to the ANOM (graphical technique), the “conclusion” was the same as the 2proportions test at alpha=0.05.
Conversely, I tried the 2proportions test with CL set to 99.7, and the CI for the difference now included zero and the pvalue was 0.014, greater than 0.003 & failing to reject the null (Fisher’s Exact Test p=0.005, still greater than 0.003). This is consistent with the first ANOM where the “control limits” were set at 3 “standard deviations”, and leads to the same “conclusion”.
Is the “consistency” in the results here a fluke, or is it expected? If it is expected, then is it true that it might not matter so much which method I choose?
Of course, then the real question is: how do I help the decision makers understand the “risk” of making a decision under uncertainty? What is the impact of making a decision? What is the risk of saying there is no difference when there is (type I) versus saying there is a difference when there isn’t one (Type II)? What is the power of the test given the sample size and chosen level of significance? oh, my head hurts…
0 
AuthorPosts
You must be logged in to reply to this topic.