iSixSigma

Give me the power

While reading about type I & II errors and specifically beta-risk I realised that although I was happy with alpha-risk I didn’t recall seeing beta-risk in any of our sampling or hypothesis equations. Being curious, I wondered why I was only accounting for type I errors in my work?

Just to review, we need to be aware of and manage our risks when running a hypothesis test. We term the potential errors as Type I & II and they can be quite insidious:

  • Type I – False Positive – You call a difference when there isn’t one
    On March 23rd 1998, Professors Fleischmann & Pons announced they had achieved sustained thermonuclear reactions in a test tube. They believed they had created Cold Fusion. Unfortunately it was not to be.
  • Type II – False Negative – You do not call a difference when there is one
    The effects here can be devastating, such as releasing harmful new drugs or missing production defects in car-tyres. Imagine a control chart showing no issues while bad product is being pumped-out?

Our (abbreviated) process for hypotheses testing being:
a. Define Ho
b. Set the alpha-risk (normally 95%)
c. Calculate p-value for the hypothesis
d. Accept or reject Ho

No mention of beta-risk here. I reviewed assorted materials to get up to speed and discovered all about beta-risk. Here is the concept (not the math), big disclaimer, I might be wrong or have missed something.

Imagine you draw a sample from a process and get an estimate for the mean and confidence interval. You decide the process needs a touch of improvement. You draw a sample from the new process and get the mean and confidence interval. Where the beta-risk comes into play is the when looking at the potential degree of overlap of the second sample with the first.

Handpicked Content:   To p Or Not To p

In the first picture (below) the red & green lines show the respective samples 95% confidence intervals. The two sample distributions are miles apart. Not much beta-risk here.

Advertise on iSixSigma.com
Advertisement

In the second picture (below) the two distributions overlap. So for any given sample from B there is a probability it will be in the zone to the left of the red-line, the 95% confidence interval of sample A, hence accepting Ho when the two means are different.

The shapes of the sampling distributions are governed by the sample size, alpha and standard deviation and have not given you the resolution to see the change in mean. The distributions are too wide. So the beta-risk is the probability of drawing a samplefrom Bin the region left of the red-line, within the 95% interval of sample A.

You can find the beta-risk for any given hypothesis test in Minitab under Stats–>Power & Sample Size. If you find you have a low score, The Power of the Test, you may wish to change your key parameters, sample size, and alpha, to improve the sampling resolution.

My conclusion on why beta-risk does not appear more prominent in our manuals is because we look for “big improvements”. If our process improvement has shifted the mean by a small-margin (so that beta-risk applies) then its not much of an improvement. Bit of a generalisation and am sure there must be exceptions.

Comments 3

  1. Jed Rothwell

    You wrote:

    "They believed they had created Cold Fusion. Unfortunately it was not to be."

    That is completely incorrect. Fleischmann and Pons were replicated by researchers in hundreds of world-class laboratories such as Los Alamos and BARC. Thousands of papers describing these replications have been published in peer-reviewed journals and conference proceedings, and in official publications from BARC, the U.S. Navy, Tsinghua U. and other institutions. I suggest you read some of this literature before commenting on cold fusion. You will find a bibliography and more than 500 full text papers here:

    LENR-CANR.org

    – Jed Rothwell
    Librarian, LENR-CANR.org

    0
  2. Jed Rothwell

    You wrote:

    "Wow, is this Jed Rothwell, author of ’Cold Fusion Still Hot’.”

    Yup, that’s me. I’ll be darned. I did not know anyone was selling that.

    ". . . and translator for Amazon 5-star rated, “Nuclear Transmutation: The Reality of Cold Fusion’?"

    Well, the stars are for Mizuno. I’m just the translator. I am glad to see it is well rated.

    "My memory of events back then was from the standpoint of an interested member of the public rather than an active scientist. So I recalled here elements of how the press reported the story including how the story was covered by a number of major magazines"

    You are right. The press back then reported that the claims were mistaken. They still do, and most newspapers report that the effect was never replicated. However this has no bearing on what actually happened. You have to read original source scientific papers to learn that.

    "Also how other researchers first agreed with and then withdrew their support for the experiment."

    I don’t recall many who did that. There was one group at Georgia Tech who made a claim and then retracted. Several groups could not replicate. However, by September 1991 there were 92 fairly-solid replications. They have never been retracted and by now there are hundreds of others.

    "Given your much closer understanding of this area it would be good if you could provide a quick update to ensure people have the true facts."

    It is hard to write a quick update. It is complicated research. I suggest people should read original source papers. Those papers are difficult to understand. I guess I would say that if you can’t understand them, then you should not try to judge the issue.

    Anyway, I do not think your original claim that "unfortunately it was not to be" is accurate.

    0
  3. Robin Barnwell

    Wow, is this Jed Rothwell, author of “Cold Fusion Still Hot” and translator for Amazon 5-star rated, “Nuclear Transmutation: The Reality of Cold Fusion”? I must say it’s great to have you critique my work.

    My memory of events back then was from the standpoint of an interested member of the public rather than an active scientist. So I recalled here elements of how the press reported the story including how the story was covered by a number of major magazines

    Also how other researchers first agreed with and then withdrew their support for the experiment. Given your much closer understanding of this area it would be good if you could provide a quick update to ensure people have the true facts.

    Regards
    Robin

    0

Leave a Reply