Test to See if Percent Change Is Significant?
- August 3, 2018 at 7:54 am #56059
I have two columns of raw data and have calculated the percent change in a 3rd column.
I ran a unmatched t-test on the raw data (men/women) p < .0005. Is there a test that addresses if the percent of change is significant?
SteveAugust 3, 2018 at 9:11 am #202891
Since you only have a single percentage change the only way to assess significance is to have something to use as a yardstick. So the question becomes significant compared to what?August 6, 2018 at 7:09 am #202896
I’m not sure if you are confusing statistical significance with practical significance, or if you are looking for a paired t test.
Statistical significance means that the difference you observed between your sample groups is too big to be reasonably explained by sampling error (chance). Based on your t test, the difference is statistically significant. Calculating the difference as a percent change only changes the way you are looking at it, not the outcome of the test — it is still statistically significant. If you are looking for practical significance (is the difference big enough to be relevant to your situation), there is no statistical test for that. You’ll need to make that decision using other criteria.
On the other hand, if you are asking how to test the change between paired groups (each member of one group has a direct association with one member of the other group), there are two ways to do that. You can either use a Paired t test on your original two columns of data, or you can use a 1 sample t test on a calculated column, setting the null hypothesis to 0 (zero). The calculated column can be just the difference between each matched set or the percent difference – either will work. Both the Paired t test and a 1 sample t test on the calculated differences will give you the exact same result. I can also tell you that if a 2 sample t test is significant, a Paired t test on the same data will always be significant.August 6, 2018 at 4:55 pm #202899
@marson You should have been taught that when they taught you to do the test. It is part of your sample size selection.August 6, 2018 at 5:19 pm #202900
This question is not related to sample size. I am seeking a statistical test to assess percent change. I am using national demographic data.August 8, 2018 at 9:15 am #202903
Again, if you just want to test the percent change in the overall average between two groups, your original t test did that. The only difference between the measured change and the percent change is the units.
If you want to test the average percent change between individual pairs, you can use a Paired t test (or a 1 sample t on the calculated differences as I described above).
If you are looking for something altogether different, please clarify exactly what you are trying to test, including your null and alternate hypotheses, along with an explanation of what your two data sets are.August 8, 2018 at 10:31 am #202904
@jazzchuck, I could be wrong but I think the OP is after what I commented on in my first post. He/she has, in some manner, computed a percent change and is looking for significance with respect to that difference where the comparison of the generated percentage change is with something external to the data.
The issue isn’t that of difference between the groups – as you and others have noted – that information is already part of the t-test. As written the OP does not seem to have anything that would justifying pairing individual measures so a paired comparison would be of no value. In any event the question usually addressed by pairing – is the mean of the differences significantly different from 0 – does not appear to be of interest to the OP.
I think if the OP provides a description of how they generated the estimate of percent change someone might be able to offer some insight with respect to answering the original question.August 8, 2018 at 2:22 pm #202905
@rbutler, you may be right. The reason I thought a paired t test might apply is because @marson stated that he calculated the percent change in a 3rd column. Since he referred to a column and not a cell, it seems plausible that he has a separate calculation for each row of data, which would suggest pairs.
Either way, some clarification would definitely help to understand the problem.August 8, 2018 at 2:32 pm #202906
I did a t-test. P < .00005 Good Stuff!
I was wondering if there was anything I could do to statistically test the “percent change” I can’t find anything and I am beginning to believe that there is no test that can address this issue.August 9, 2018 at 4:59 am #202908
There are ways to test the significance for a percent change the problem is that before anyone can offer anything you still need to tell us what you have done and how you want to test the percent change.
1. You said you have a 3rd column with a percent change – how did you do this computation? For example:
a. did you just put male and female columns side-by-side, take a difference and compute a percent change using one of the two columns as a reference?
1. If you did a) then what was your criteria for pairing the two columns? If you just randomly smashed the two columns together and took a difference then the percentage changes are of little value.
Or did you use some other method. Regardless, we need to know the computational method you used and what is being compared to what.
2. You said you want to know if this percentage change is significant. Based on your earlier posts it appears that you do not want to test to see if it is different from 0 therefore, what is the target? That is, what target percent change do you want to use for comparison?
Example: you have a precious metal recovery system that recovers 99.986% of all of the metal from scrap. You have a new method which you believe will recover 99.990% and you have a target for significant change of .002% – under these circumstances if the new process delivers as expected the percent change will be significant.
You must be logged in to reply to this topic.