You did not specify the test that the p-value is associated with. It sort of sounds like you are talking about a test of normality.
There are several goodness of fit tests for normality – probably the most common are the Shapiro-Wilke Test (the Ryan-Joiner Test in MINITAB), the Anderson-Darling A^2 Test, the Kolmogorov-Smirnov test, and the chi-square test.
The formulas are more complex than can be presented in this format, but they are all covered in Ralph B. D’Agostino & Michael A. Stephen’s book “Goodness -of-Fit Techniques”.
The Shapiro-Wilks test uses the correlation observed in the normal probability plot. The Anderson-Darling and Kolmogorov-Smirnov tests both compare the empirical cdf to that of the best fit normal curve. The Anderson-Darling test pays more attention to the tails than does the K-S test.
D’Agostino & Stephen’s recommendations are that the Shapiro-Wilks test and Anderson-Darling A^2 test are amoung the best to use. They indicate that the Shapiro-Wilks type tests are probably overall most powerful. MINITAB defaults to the Anderson-Darling test, but I don’t really know why.
D’Agostino & Stephens go on to say:
“For testing for normality, the Kolmogorov-Smirnov test is only a historical curiosity. It should never be used. It has poor power in comparison to the above procedures.”
and
“For tetsing for normality, when a complete sample is available the chi-square test should not be used. It does not have good power when compared to the above tests.”
.
