How Much Confidence is Enough?

Six Sigma – iSixSigma Forums Old Forums General How Much Confidence is Enough?

Viewing 14 posts - 1 through 14 (of 14 total)
• Author
Posts
• #48310

Ryan
Member

I’m running a binomial problem and have a result that says I’m 85.5% confident that there is a difference between two items.  The usual test is for 90 or 100% confidence.  But at what point do we say, “nope, 85.5% sure is not sure enough?”  Most people I talk to (who are not statisticians) consider 85.5% sure enough proof of difference.  How do you all feel about a judgement like this?
Ryan

0
#162446

Mikel
Member

This is something you want to decide before the test – so that you will make sure you had an adequate sample size to show that difference if it indeed exists. How big of a difference is considered important? How big of a sample did you have?

0
#162452

Ryan
Member

Stan,
I know what you’re saying, as those are the standard steps for conducting statistical tests.  My point was more along the lines of explaining results to managers who has very little knowledge of probability and statistics.  My single biggest frustration in dealing with organizational managers is their entrenched reliance on their own gut feel, even in defiance of highly-significant statistical data that may point in the opposite direction of their guts.  I have seen evidence where there was 99% confidence ignored, but more often it is the other way around.  Managers will see a “spike” and make rash assumptions about a difference, and frequently waste time and money acting on this “difference,” even when the statistics are screaming that it was random variation, and we can not rationally conclude that there was any difference.  This just got me thinking about our ability to translate our findings to decision-makers.  I imagined myself in one of those meetings with “the boss,” where I report that based on our tests, the results from December do not represent a significant difference from the other months of the year.  And the boss says, “how big a difference would this have to be to conclude it was significant?”  And I say that it depends on our confidence level.  And he says, “so you’re telling me that this WOULD be a difference if I wanted to be 85% sure, but it ISN’T a difference if I want to be 95% sure?  I can see this going very badly downhill.  And frankly, I got to thinking that sometimes 85.5% would be good enough to conclude a difference.  Or would it?
That was what was going on in my head when I posted my question.  I don’t want to say that bosses are morons to ignore accepted scientific data, but…….
Oh well, sorry to be long-winded.
Ryan

0
#162453

Darth
Participant

Ryan, confidence is one aspect. I would explore what your Power is as well. Power is the ability to spot a difference if one exists. That may be as important if not more than the confidence. If you end up rejecting the null, then power is moot. But, if you don’t reject the null then it might be due to the fact that no difference is there or that the power of your test is insufficient to spot the difference.

0
#162455

Snow
Participant

I would say just enough to get the number but not enough to warrant a drink in the face or extensive verbal abuse.

0
#162456

Szentannai
Member

Hi Ryan,
I think rephrasing the statement in terms of risk could help you.85% percent confidence means you accept a risk of 15% of being wrong if you act based on the data, 95% means a smaller risk of 5% of being wrong.So, what you tell your manager makes very good sense: you are telling him to act if he is happy with 15% risk and to not act if he only accepts 5% risk or less. BTW there is no such thing as 100% confidence :) not based on sampling and statistics.Kind regards
Sandor

0
#162458

Taylor
Participant

Ryan, do you work for Tyco, sounds very familiar

0
#162461

Mikel
Member

No problem with the long winded stuff, we have all been there.
My advice is go back and look at sample size, if it were adequate to show a difference with 95% confidence, don’t accept 85%. If on the other hand you needed a lot more samples than you have backing of to 85% is rational.
On the talking to the managers. If they are smart enough to ask about confidence, then tell them and they will agree with your conclusion. If they aren’t smart enough, recognize that and don’t talk stats to them. One of the worst things a BB or GB can do it give a presentation to management with all sorts of stats in it – keep it as back up – if asked.

0
#162463

Dr. Scott
Participant

Ryan,
Whether make a process change with 85% confidence that it will result in Y improvement is really a personal decision. How much of a risk taker are you and your decision makers? I have bet on worse horses and lost (but have also won). Another thing to consider is whether the change is easy, quick, and cheap? If all three, then it might be worth a try.
However, my favored advice would be similar to what Stan said. If you have been able to get the data from the two groups to this point, why not simply continue collecting the data between the groups until you can be more confident it you findings one way or the other?
As far as your problems with management, this is nothing new. They need to come to understand the use of your skills to help them make decisions. Perhaps they themselves need a bit of training so they understand what it is you are telling them. But if you can get a few big successes under your belt (no pun intended), then I think their attitude will more quickly change.
Good Luck,
Dr. Scott

0
#162464

Dr. Scott
Participant

Ryan,
Stan is leading you down the right path. Perhaps the two of you need to spend more time on both the stat question and the management question.
Either can come back to me if you need ideas on either.
Dr. Scott

0
#162471

Ryan
Member

Thanks for all the replies everyone!
Ryan

0
#162473

Ryan
Member

No I don’t work for Tyco:).
Ryan

0
#163105

M K Rana
Participant

I think the confidence level is dependent on the sample size and also the seriouiness of your decision. If the impact of your decision is great than you should take decision at higher confidence level otherise it is OK.
In the start itself you should decide the lvel ofconfidence for decision so that you can increase the sample size to increase the confidence level.
bye

M K Rana

0
#163111

psyc20
Participant

I would agree with all the points and would consider specifically what M K Rana and Dr. Scott said about Quick, cheap changes and risk.
What is the result if you are wrong? Would people get injured or die? Would the cost of the change bankrupt the company? This should have been discussed during the planning portion of the trial. Have you completed an FMEA? This should highlight potential failures and there severity.
If the impact of an incorrect decision in minimal then try it, if not get more data before making changes.
This may not be the statistically preferred response but it has worked for me in the past.

Psyc

0
Viewing 14 posts - 1 through 14 (of 14 total)

The forum ‘General’ is closed to new topics and replies.