Choosing the Right Hypothesis Test
- This topic has 8 replies, 5 voices, and was last updated 5 months, 1 week ago by
.
As you suggested I plotted yes/no vs percentage and came up with the graph that I attached here. But I can’t conclude any regression from this plot. Is this the plot you meant?
this is the explanation from minitab; alpha which is the risk of concluding that the mean of Yes differs from the mean of No when in fact it doesn’t, is .05. if the p-value is less than .05, you can conclude that the mean of Yes differs from No at the .05 level of significance.
I kind of don’t understand why. we calculate t-value then transfer that into p-value. when the p-value gets bigger the t-value gets smaller. and for a smaller t-value the probability that we say there is no difference while there is difference gets higher. what am i missing here?
It’s not a matter of calculating the t value and then transferring it into a p-value. What you are doing is computing a t statistic and then checking that to see if the value you get meets the test for significance. Small t-values breed large p-values and conversely. :-)
You are doing what I recommended and your plot is telling you what you computed – there is a slight offset between the two groups of data which means that the averages are numerically different but there really is no statistically significant difference. If you want an assessment of the means then you could turn the analysis around – treat the yes/no as categorical and run a one way ANOVA on the results. You’ll get the mean values for the two choices and you will also get no significant difference.
All of the above is based on commenting on the overall plot you have provided. There are a couple of interesting things about your plot you should investigate. It is a small sample size and it looks like your lack of significance is driven by 4 points – the two points at -1 that are much lower than the main group and the two points at 1 that are higher than the main group.
As a check – remove those 4 points and run the analysis again – my guess is you will either have statistical significance (P < .05) or be very close. If you do get significance, and if it is a situation where you were expecting significance, then you will want to go back to the data to see if there is anything out of the ordinary with respect to those 4 points. If there is something that physically differentiates those points from their respective populations and if their deletion results in significance then I would recommend the following:
1. Report out the findings with all of the data.
2. Include the plot as part of the report.
3. Emphasize the small size of the sample.
4. Report the results with the deletion of the 4 points.
5. Comment on what you have found to be different about the 4 points.
6. Recommend additional samples be taken with an eye towards controlling for whatever it was that you found to be different about the 4 points and re-run the analysis with an increased sample size to see what you see.
You must be logged in to reply to this topic.
