# P-Value of 0.05, 95% confidence

Six Sigma – iSixSigma › Forums › Old Forums › General › P-Value of 0.05, 95% confidence

This topic contains 33 replies, has 23 voices, and was last updated by Sam Oryx 9 years, 7 months ago.

- AuthorPosts
- June 14, 2002 at 3:10 am #29645
I always have a doubt here. In hypothesis testing, when the p-value is > 0.05, we accept the null hypothesis and the alpha risk is 0.05 (95% confidence). What about if i want 90% confidence? Does it mean that I can accept the H0 if p-value is > 0.10 if i want 90% confidence? It does not really make sense to me. please advise. thank you.

0June 14, 2002 at 8:09 am #76409

James AParticipant@James-A**Include @James-A in your post and this person will**

be notified via email.You are not alone – stats is not my forte, either. The purist(s) will probably disagree, but I find it easier to view the ‘p’ as the probability of your hypothesis being right or wrong – this then must mean that as one probability goes up, the other reduces, as your total probability has to remain ‘1’ to account for 100% of everything you may expect to see.

So yes, for your example of 90% confidence, you can accept H0 if p>0.1

Hope this helps and does not result in my being burnt at the stake.

James A0June 14, 2002 at 12:39 pm #76417Think of the p-value as a percentage, ie. p-value of 0.05 =5%. p-value of 0.13 = 13%. The p-value is then the percentage chance of being wrong if you reject the null.

David0June 14, 2002 at 2:36 pm #76426

GabrielParticipant@Gabriel**Include @Gabriel in your post and this person will**

be notified via email.Please, someone with a good “hypothesis tests” knowledge read and correct this if I am wrong.

If your p < 0.05 and you REJECT the null hypothesis, then you have a risk of 5% of having rejected it when it had to be accepted, or a 95% of confidence of having rejected it correctly. This is the "Alpha Risk".

If you ACCEPT the null hypothesys because p>0.05 then forget about Alpha Risk (Alpha Risk, the risk of wrongly rejecting the null hypotesys, exists only when the null is rejected). You have a 95% confidence of nothing. But you have a “Betta Risk” of having accepted the null when it had to be rejected, or a (1-Betta Risk)x100% of confidence to having accepted it correctly. I don’t know exactly how does the betta risk work, but I think that is a function of the type of hypothesis test, p and sample size. What I remember is that once the test is selected and the alfa risk (maximum P to reject the null) is chosen, then the only way to improve (reduce) the betta risk is to increase the sample size.

Shortly, if you test a null hypothesys with an alpaha of 5%, then you can be 95% sure that if you find a p<0.05 the alternative hypothesis is true, and then the null is false. If you dont find a p<0.05, then the only thing you can say is that you are not 95% sure that the null has to be rejecetd, that is not the same that saying that you are 95% sure that the null must not be rejected. This would be the betta risk.

Please, others. Was that correct?

Gabriel0June 14, 2002 at 4:02 pm #76429One way to look at this is by seeing the distribution curve for your data. If you have a 95% confidence level, then your answer means that it lies within the USL and LSL for this curve. The USL and LSL would each be at 2.5% at each end of the curve.

For a reference you can look in Breyfogle’s book, “Implementing Six Sigma”, page 54.

Hope the is helps.

TG0June 14, 2002 at 4:40 pm #76430The rejection of a hypothesis at the 5% level does not imply that the probability that the hypothesis is false is 95%; it implies that the observed result belongs to a class of results whose overll probability of occurrance, if the null hypothesis is true, is 5%.

This provides good reasoning for supposing the hypothesis is false, but no numerical value can be placed upon this degree of belief.

That is one reason why you should calculate a confidence interval as well.0June 14, 2002 at 6:20 pm #76436

GabrielParticipant@Gabriel**Include @Gabriel in your post and this person will**

be notified via email.Thanks for your calrification.

It is alittle bit confusing… Some times (specially in statistics) there are some statements that seem to be equivalent but are not. I understand what you explain, but then I have new doubts.

If we reject the null hypothesis at a 5% level, are we 95% confident of something?

Once rejected, is 5% the risk (or probability) of having rejected a true null hypothesis? It is not, according to you. What is, then, alpha risk?

And what about if we can’t reject the null hypothesis at a 5% level? What can one infer from that?

I would appreciate if you could put some more light on the subject.

Thanks

Gabriel0June 14, 2002 at 6:48 pm #76440

Mike CarnellParticipant@Mike-Carnell**Include @Mike-Carnell in your post and this person will**

be notified via email.Gabriel,

I think the concept you are looking for is called the power of the test. There is some stuff on it in Design and Analysis of Experiments by Douglas Montgomery page 21- 22 (I have the second edition so it may be old and the page numbers may be off).0June 14, 2002 at 7:40 pm #76442Gabriel,

The 5% (alpha risk) is the probability of a type I error – rejecting the hypothesis when it is true. This has little to do with the beta risk, which is the probability of a Type II error, the probability of not rejecting the null hypothesis when it is false.

Mr. Carnell’s comments about the power of a test should help you with this issue. Power = 1-beta risk — in other words, it is the probability of correctly rejecting the null hypothesis when it is false.

Montgomery says: “The general procedure in hypothesis testing is to specify a value of the probability of a Type I error alpha, then to design a test procedure so that a small value of the probability of Type II error beta is obtained.

“We speak of directly controlling or choosing the alpha risk. The beta risk is generally a function of sample size and is controlled indirectly. The larger the sample size, the smaller the beta risk (the risk of not rejecting a false hypothesis)”

0June 14, 2002 at 11:21 pm #76444

GabrielParticipant@Gabriel**Include @Gabriel in your post and this person will**

be notified via email.Ben:

Now I am not sure, but I think I understand (and undertood before) what alpha risk and betta risk are. Your definitions seem to me the same that those I provided in my first post. Apparantly, my mistake was to say that the level (i.e. 5%) was the probability for the null hypotesis to be true if you rejected it after finding a p<0.05. Now, I understand that there must be something wrong in my reasoning, but I can't find what:

I make a test at a 5% level. I find a p<0.05 so I reject the null hypotesis (because the level was set at 5%). Then I have an alpha risk of 5%. Then the probability of having rejected a true null hypotesis is 5%. But I actually rejected it. So there is a 5% of probability that the null hypotesis is true. Or, what is the same, a 95% of probability that the hypotesis is false (P(true)+P(false)=100%)

But you said “The rejection of a hypothesis at the 5% level implies that the observed result belongs to a class of results whose overall probability of occurrance, if the null hypothesis is true, is 5%. This provides good reasoning for supposing the hypothesis is false, but no numerical value can be placed upon this degree of belief”.

Again, I uderstand what you say and it seems correct to me, only that I can find what is wrong with my reasoning, and your explanation and my reasoning can’t be both true. How does the “level” (probability of occurrance of a result in a class if the null hypotesis is true) relates with “alpha risk” (probability of rejecting a true null hypotesis)? Up to now I thought that “level”=”alpha risk”. Was at least a part of my first message correct? If not I should forget that I once studied Hypotesis Testing and start all over again.

Would you make a last effort to get me out of this mess?

Thanks

Gabriel0June 17, 2002 at 5:26 pm #76472The basic answer to your question is Yes, if you specify an Alpha risk of 0.1 then if the p-value is greater than 0.10 you would accept the null and conclude that there is insufficient evidence to show a difference, and conversely if the value were < 0.1 you would reject the null.

However, what does that really mean is the second part of the question. An alpha test assumes “Truth” to be that the null hypotheses you establish is correct and you are assigning the risk of being wrong (a type I error) and saying as an example, there is a difference in the parameters when in fact there is not.

This can be seen in the simple t test (and then expanded to other test) where you have the means of two samples and say you are testing the hypothesis that they are the same (come from the same population). Given one of the samples, you can determine the confidence interval (say at 95%) for the mean based on the number of samples and the variation within the sample. If you remember what this means is, that if you were to take another sample from the same population, you would expect to have the new samples mean to fall within that interval 95 out of the next 100 times, or that the new sample mean would be outside that interval only 5% of the time. This basically is the hypothesis test you get the second sample and if its outside the interval you would conclude that it comes from a different population (that the means are different) and if you do that, and “Truth” is that they are the same, you have a chance to be wrong and the new mean might just happens to be one of those samples where you wouldnt expect it to be. You have made then a Type I error.

Now what happens if you increase the risk of being wrong (going from alpha being 0.05 to 0.1)? The confidence interval for the 1st sample gets smaller. In other words, 10 times out the next 100 samples you would get a mean that you would not expect from the sampling the same population. Equivalently in the hypothesis test, you would reject the null that the samples are the same when in fact, they are, more frequently again a type I error.

Finally, how does the p-value figure in to all this? The p-value is basically the percentage of times you would see the value of the second mean IF the two samples are the same (ie from the same population). The comparison then is in the risk you are willing to take in making a type I error and declaring the population parameters are different. If the p-value is less than the risk you are willing to take (ie <0.05) then you reject the null and state that with a 95% level of confidence that the two parameters are not the same. If on the other hand, the p-value is greater than the risk you are assuming, you can only tell that there isnt enough difference within the samples to conclude a difference. Where you set your risk level (alpha) then determines what p-value is significant.0June 17, 2003 at 11:21 pm #87052I am very new to sixsigma, so here are my ? If I have a p-value of 0.00 in my descriptive stats, what does that mean. What is the box plot under my histogram tell me? Is the 95% confidense interval for Mu mean that 95% of the population mean will fall into to that range?

0June 18, 2003 at 5:40 am #87058Gabriel

In my understanding u r right. They r not the same.

Situation : p value is > 0.05 means…

1. u r < 95 % sure if u reject the null

2. u r > 5 % sure if u do not reject the null

3. u r > 5 % sure if u accept the null

4. u r < 95 % sure if u accept the alternate hypo

In case the situation demands a 90 % confidence level then the abv figures become 90 % i/o 95 % and 10 % i/o 5 %

prasad0June 18, 2003 at 12:47 pm #87070Yes! That is exaclty correct.

Ron0June 18, 2003 at 1:22 pm #87076

faceman888Participant@faceman888**Include @faceman888 in your post and this person will**

be notified via email.Gabriel,

Don’t forget it all and start all over. Go to this link, it gives a graphical representation of alpha and beta. Read it. Figure four shows the beta probability in yellow and the alpha in red.0June 18, 2003 at 2:00 pm #87078

faceman888Participant@faceman888**Include @faceman888 in your post and this person will**

be notified via email.Gabriel,

Sorry. I forgot to attach the link:

http://www.intuitor.com/statistics/T1T2Errors.html?cfA3F3228E=YTIzMjEwODpnb29keWVhcmludGVybmV0OqxFzpaudgM/kcAEooCir9E=0June 19, 2003 at 10:56 am #87099

HemanthParticipant@Hemanth**Include @Hemanth in your post and this person will**

be notified via email.Hi Adrian

In a hypothesis test you either accept the null hypothesis or accept the alternate hypothesis.(as somebody here said..) p value is “the probablity of null hypothesis being true” and 1-p is nothing but probaility of alternate hypothesis being true.

You are right about the p values. Intuitively, if you wish to be less confident (i.e confidence level) in accepting the alternate hypothesis then your p value will have to be large enough so that probablity of your alternate hypothesis being true (i.e 1-p) is less than the confidence level.

Well, you sure can question the logic because you are accepting the null hypothesis even when it has just 5 or 10% chance of being true but rejecting the alternate hypothesis even when there is 95 or 90% chance of it being true. Hypothesis tests are that way more weighed towards accepting the null hypothesis. The premise is I would not accept any difference until I dont see a very strong indication ofor existence of such a difference. hope this makes sense..0September 6, 2003 at 5:44 am #89624

Muhammad A khanParticipant@Muhammad-A-khan**Include @Muhammad-A-khan in your post and this person will**

be notified via email.Gabriel,

I have a confusion in your previous post in which you said “….. What I remember is that once the test is selected and the alfa risk (maximum P to reject the null) is chosen….” My question is that what is the criteria of selecting the alpha risk, how can we say that 5% risk is acceptable, not 10%?? Is this decision based on experience?

Thanks for help & regards0September 8, 2003 at 3:06 am #89646

Muhammad A khanParticipant@Muhammad-A-khan**Include @Muhammad-A-khan in your post and this person will**

be notified via email.Gabriel, I am sorry I am late asking this question. It is the continuation of the thread ” P-value of .05, 95% confidence”.

I have a confusion in your previous post in which you said “….. What I remember is that once the test is selected and the alfa risk (maximum P to reject the null) is chosen….” My question is that what is the criteria of selecting the alpha risk, how can we say that 5% risk is acceptable, not 10%?? Is this decision based on experience?

Thanks for help & regards0September 8, 2003 at 12:22 pm #89664

GabrielParticipant@Gabriel**Include @Gabriel in your post and this person will**

be notified via email.I am not sure, I don’t think there is one criterion or “formula” you can apply and get the “right alpha risk to use”. It’s a trade-off between alpha risk, betta risk, sample size, cost of sampling and testing, and the impact a wrong decission may have (either if the null is wrongly accepted or if the null is wrongly rejected)

The “default” value seems to be 5%. It seems to be a common practice to use 5% unless you have a reason to use another value. Maybe this figure gives a good balance between the many factors involved.

Sometimes 10% is used (difficult to gather data, expensive testing, low impact of a wrong rejection, high impact of a wrong “fail to reject”). Other times 1% is used (lots of data readily available, high impact of a wrong rejection, low impact of a worng “fail to reject”).

These values (1, 5 and 10%) are the ones most frequently seen as “decision limit”, with 5% being the number one choice.0April 7, 2004 at 12:32 am #98093Greetings ,

Just to clear my confusion , i also read (in my learning material ) that as the degree of freedom goes down the power of the test increases (Now sample size has to decrease to decrease the degree of freedom ! ) . AND here we are reading that the power of the test increase with the increase of sample size ???

Please explain what may be wrong in my inderstanding or have i read anything wrong !

0April 7, 2004 at 11:45 am #98114

GabrielParticipant@Gabriel**Include @Gabriel in your post and this person will**

be notified via email.Can you reference your learning material, so I never try to learn from it?

No, it’s a joke. What I mean is that there must be something wrong either in the material or in your understanding of it. The power improves as the sample size (or the degrees of freedom) increases. Unless I am missing something….

Would you quote exactly what your material sais?0February 15, 2005 at 1:22 am #114874I have a question in relation to the z-score.I know that alpha 0.05 is equal to the z value of 1.96, but I don’t know how that was found.

If given alpha = 0.01, how do I find its z value?

0February 15, 2005 at 1:58 am #114876Go into a Z table and find the value of .005. Look at the outside of the table and see where the values intersect, around 2.57-2.58. If you did the same for .025 you might find the 1.96. Remember the distribution is two tailed so you take 1/2 the alpha when you use the table since it is only one tail.

0February 6, 2009 at 11:55 am #180739

KarolosParticipant@Karolos**Include @Karolos in your post and this person will**

be notified via email.hello.i read the all the replies.As far as i understood the normal value of p is lower than .05 for confidence 95%. In my research i have for one factor p = .057 an for another one p = .08.

What should i do? I really need the .08-factor. Can i accept 90% confidence? and if yes who should i justify it or report it?Thanks

Karolos0February 6, 2009 at 12:08 pm #180742

Chris SeiderParticipant@cseider**Include @cseider in your post and this person will**

be notified via email.Grats on the long delay between last post date and yours….

But seriously…

Get more samples or implement change and monitor closely. My preference is get more observations and make a decision afterwards.0February 6, 2009 at 12:18 pm #180743

KarolosParticipant@Karolos**Include @Karolos in your post and this person will**

be notified via email.the topic is alive again….Its quite difficult to take more sample. the situation is complicated, is my last step to finish my master thesis. Taking more samples is out of the table:(

Either i accept the null hypothesis or i use 90% confidence.0February 6, 2009 at 2:11 pm #180749

Chris SeiderParticipant@cseider**Include @cseider in your post and this person will**

be notified via email.3rd option….

State “with X% confidence, the [means or standard deviations or medians or frequency counts or proportions] are different”. X% is 100 minus your p-value in %.

Your choice.

Good luck on you defending your masters thesis. What area of study?0February 6, 2009 at 3:03 pm #180757

KarolosParticipant@Karolos**Include @Karolos in your post and this person will**

be notified via email.HelloSo in my case is 92% of confidence. These values came out from the regression analysis i performed. This p value is the value of one factor. Do i have to do something for changing the confidence (like new calculations).

Or i just report: that for 95% significance the factor is not significant but for 92% it is?I’m really sorry about these dummy questions but i’m newbie in statistics….trying to put an order to all this mess.

My area is economics and informatics.

Thank you for your help

Karolos0February 6, 2009 at 3:09 pm #180758

Robert ButlerParticipant@rbutler**Include @rbutler in your post and this person will**

be notified via email.We’re having a similar discussion over on the thread – When to Reject a Null Hypothesis. I posted the connection to an earlier thread there. Since that earlier post applies to your question I’ll repost the link here.

https://www.isixsigma.com/forum/showmessage.asp?messageID=1451210February 6, 2009 at 3:09 pm #180759

Chris SeiderParticipant@cseider**Include @cseider in your post and this person will**

be notified via email.If your thesis adviser allows…

I’d say “with 92% confidence” a relationship is found between your input(s) and the output. Just remember you cannot assume a cause and effect relationship.

One minus your p-value gives you your confidence. Most people want at least 95% confidence so they want the p-value to be less than 0.05 if a difference was detected.0February 6, 2009 at 4:58 pm #180775Hai Karolos,

If you do Regression with several X’s you generally start with all X’s that are expected to show significance. From the Regression results you will see that some are not significant (based on your data). Remove these from the model (perform the Regression without them) and the p-values of the others (also the X that holds your special interest) will also change.

Don’t forget to check the Residuals for ‘strange behaviour’ and the R-sq to see if you have used the X’s that cover the Y (if R-sq is low that could mean that you are missing significant X’s in your model).

Good luck0February 8, 2009 at 12:31 pm #180829

Bower ChielParticipant@Bower-Chiel**Include @Bower-Chiel in your post and this person will**

be notified via email.HiI’ve found an article by Sterne & Smith entitled “Sifting the evidencewhat’s wrong with significance tests?” useful. It’s freely available at http://ptjournal.org/cgi/content/full/81/8/1464. They make the summary points: -P values, or significance levels, measure the strength of the evidence against the null hypothesis; the smaller the P value, the stronger the evidence against the null hypothesis.An arbitrary division of results, into “significant” or “non-significant” according to the P value, was not the intention of the founders of statistical inference.A P value of 0.05 need not provide strong evidence against the null hypothesis, but it is reasonable to say that P < 0.001 does. In the results sections of papers the precise P value should be presented, without reference to arbitrary thresholds.Results of medical research should not be reported as "significant" or "non-significant" but should be interpreted in the context of the type of study and other available evidence. Bias or confounding should always be considered for findings with low P values.I also like the diagram they give for the interpretation of P values at http://ptjournal.org/content/vol81/issue8/images/large/zad8181464fig3.jpeg. (I don't know how to insert it into this post – any advice would be welcomed!)Best WishesBower Chiel

0September 23, 2009 at 6:25 pm #185642Hi Ted,Thanks for your thorough explanation of the alpha and P values.You have made difference in my understanding of hypothesis testing.Cheers, Sam

0 - AuthorPosts

The forum ‘General’ is closed to new topics and replies.