iSixSigma

Are my process results really any different than yours?

Six Sigma – iSixSigma Forums Industries Healthcare Are my process results really any different than yours?

Viewing 6 posts - 1 through 6 (of 6 total)
  • Author
    Posts
  • #53775

    Andrew Banks
    Participant

    Forum Members:

    I have reached the limits of my statistical knowledge and need some help. The question in the subject line is often asked, and in this case may even be appropriate. A team I completed a DMAIC project with has vastly improved their process (8-fold reduction in defect rate & one tailed 2 proportions test of before/after data yields p<0.000) and held the results for 7 months now, but I have been asked by the process owner if the new process performance is the "same" as the rest of the hospital. This is not a metric the project planned to address, so I gave no thought as to how I would do this analysis upfront...

    The variable in question is attribute, has 2 possible outcomes (good/bad), one trial per patient, but certainly I could envision scenarios in which the trials may not be truly independent or have the same probability for success.

    So, I was able to get the last 6 months of data for this process for the entire hospital in summarized form (events/trials).

    Now, how to test my hypothesis? Should I:

    1. consider the entire hospital the “population” and use a 1-proportion test (is it really the population)
    2. subtract my team’s department from the entire hospital data and then run the 2-proprtions test (the sample sizes differ by a factor of 10 and I’m not sure I can reasonably get the raw data to take random samples)
    3. or what about using an ANOM technique where I use a p-chart to determine if either category is “different” than the overall using 3 “standard deviations”?

    I have tried all 3, and the results lead to different conclusions (well, option 3 does anyway). So, which is the most appropriate interpretation of the situation (and therefore which test is most valid)? I have been tempted to ask, “why does the answer to this question matter anyway”?, but I know that at least in part the process owner is attempting to prioritize this metric against others for next projects, which is important to me.

    Thanks for your help / input.

    0
    #191424

    Robert Butler
    Participant

    I’d recommend going with #2 and, if your software will produce them, generate the 95% confidence intervals for your proportion so that you can show where the proportion for the rest of the hospital falls relative to those intervals. If it is available, you could run the same thing but with a comparison against national standards.

    I had to do something very similar to what you are describing about two weeks ago and by showing the location of the rest of the hospital as well as the location of the national rates relative to the confidence intervals of the proportions we had generated I was able to show that, in spite of our very small samples (the issue was that of a rare occurrence of an event), the data did indicate a clinically significant improvement. I also ran a post-hoc power test on the proportions. The power wasn’t 80% but it was in the mid 60’s and, given what we had to work with, it was/is very encouraging.

    0
    #191425

    Andrew Banks
    Participant

    Robert:

    thanks for the input: I have followed your advice, and while the result is not what the process owner had hoped for, I can adequately describe the gap that still exists.

    Regards,

    0
    #191434

    Severino
    Participant

    Robert Butler wrote:

    I’d recommend going with #2 and, if your software will produce them, generate the 95% confidence intervals for your proportion so that you can show where the proportion for the rest of the hospital falls relative to those intervals. If it is available, you could run the same thing but with a comparison against national standards.

    I had to do something very similar to what you are describing about two weeks ago and by showing the location of the rest of the hospital as well as the location of the national rates relative to the confidence intervals of the proportions we had generated I was able to show that, in spite of our very small samples (the issue was that of a rare occurrence of an event), the data did indicate a clinically significant improvement. I also ran a post-hoc power test on the proportions. The power wasn’t 80% but it was in the mid 60’s and, given what we had to work with, it was/is very encouraging.

    Really? I would’ve said #3. Isn’t the purpose of an ANOM to compare the average of different groups to the overall average and determine if a significant difference exists? Isn’t this exactly what he was asked to do or am I just stupid?

    0
    #191440

    Robert Butler
    Participant

    Yes, the object of ANOM is to compare various group means/proportions to their overall mean/proportion but the request, as stated in the first post was to see, “if the new process performance is the “same” as the rest of the hospital.”

    My interpretation of this request is that the desired comparisons is that of two proportions (the hospital proportion less the group of interest and the particular group proportion) relative to one another and not that of taking the entire hospital, computing an overall proportion, and then running an analysis of all of various hospital groups, including the group of interest, against a grand proportion.

    0
    #191441

    Andrew Banks
    Participant

    Robert:

    I’m a little confuddled (yes, that’s a word of my own creation). I think I’ve reached a conclusion, but I could be way off. Any help getting out of the mire is greatly appreciated…

    I know that a 2-proportions test and the ANOM technique are fundamentally different (testing the independence of two binomial variables versus comparing them both to the overall average), but does it make any sense that they should “roughly” agree (i.e. analogous to the 2-sample T and ANOVA) when the number of proportions being compared is reduced to 2 (and the test difference,H0, is 0)? Well, that was the thought I went with anyway.

    My software (Minitab) offers 2 tests for 2-proportions (one based on normal approx. and Fisher’s Exact Test). The ANOM I created by summarizing the data (defects,trials) in each “category”, (36,2724) and (174,22682) and then created the p-chart.

    I stated in my first post that the ANOM led me to a different conclusion. That was partly in error – I had not set the “control limits” at the alpha=0.05 level (roughly 2 “standard deviations”) to match the 2-proportions test. When I made this adjustment to the ANOM (graphical technique), the “conclusion” was the same as the 2-proportions test at alpha=0.05.

    Conversely, I tried the 2-proportions test with CL set to 99.7, and the CI for the difference now included zero and the p-value was 0.014, greater than 0.003 & failing to reject the null (Fisher’s Exact Test p=0.005, still greater than 0.003). This is consistent with the first ANOM where the “control limits” were set at 3 “standard deviations”, and leads to the same “conclusion”.

    Is the “consistency” in the results here a fluke, or is it expected? If it is expected, then is it true that it might not matter so much which method I choose?

    Of course, then the real question is: how do I help the decision makers understand the “risk” of making a decision under uncertainty? What is the impact of making a decision? What is the risk of saying there is no difference when there is (type I) versus saying there is a difference when there isn’t one (Type II)? What is the power of the test given the sample size and chosen level of significance? oh, my head hurts…

    0
Viewing 6 posts - 1 through 6 (of 6 total)

You must be logged in to reply to this topic.