5 Estimation, hypothesis tests and clinical significance
Estimation (using confidence intervals)
Assuming that the null hypothesis is true, a p-value indicates the likelihood of obtaining results at least as extreme as those found in the study. It can only be used to decide whether the results are statistically significant or not, it does not give any information about the likely size of the difference. Much more information, such as whether the result is likely to be of clinical importance can be gained by calculating the difference and its confidence interval. As confidence intervals are so informative there is a growing consensus that only the estimate of the effect size and its confidence intervals should be reported for studies. However, it is unlikely that p-values will ever be eliminated as a way to quantify differences.
For example consider a large study comparing two treatments for high blood pressure; the results suggest that there is a statistically significant difference $$(p=0.023)$$ in the amount by which blood pressure is lowered. This p-value relates to a difference of 3mmHg between the two treatments. Whilst the difference is statistically significant, it could be argued that a difference of 3mm Hg is not clinically important. This is supported by the 95% confidence interval of 2.3mm Hg to 3.7mm Hg.
Clinical and statistical significance
A clinically significant, or important, difference is one that is large enough to make a difference to patients or patient management. It should be noted that this is usually subjective and may differ depending upon who is making the judgement. Whilst a result may be statistically significant, it may not be clinically significant (relevant/important) and conversely an estimated difference that is clinically important may not be statistically significant: absence of evidence does not equate to evidence of absence. Figure 4 illustrates this as it displays the results of 7 theoretical studies comparing two treatments. It shows a range of possible estimates of the treatment difference and their confidence intervals.
 |
Figure 4: Relationship between statistical and clinical significance |
Contents