Proper use of post hoc power analyses
When research results are not statistically significant, authors sometimes perform post hoc analyses of the power of their statistical tests. As stated previously, however, “Calculating power using observed effect sizes is not helpful because such estimates are very poor estimates of the actual power of the test.”1
Importantly, poor estimates of test power have correspondingly large 95% confidence intervals. As an example, Gordon-Evans et al,2 in a study examining long-term outcomes of dogs with cranial cruciate ligament rupture treated with either a lateral fabellar suture technique or tibial plateau leveling osteotomy, did not find a statistically significant (P = 0.06) difference between treatment groups for peak vertical force 6 months after surgery but did find a significant difference 12 months after surgery. This set of circumstances—where some results are statistically significant and others are not—represents a classic situation for post hoc power calculations.
For the dogs in this study, a post hoc power calculation for the t test comparing peak vertical force 6 months after surgery yields a power of 39.6%, but the 95% confidence interval for power ranges from 3.8% to 89.8%, indicating that 39.6% is a poor estimate of the true power because it is too unreliable. The 95% confidence interval includes 80%, the lower limit of power typically considered to be acceptable, suggesting that the original t test may have had acceptable power even though the point estimate (39.6%) was well below 80%. Regardless, the wide 95% confidence interval makes interpretation of the point estimate of power difficult.
Power analysis is an important component of study design before data are collected but generally is unhelpful in the post hoc situation. If post hoc power analyses are performed, they should always be reported with confidence intervals.
Richard Evans, phd
Missouri Orthopaedic Institute
University of Missouri
Columbia, Mo
1. Thomas L. Retrospective power analysis. Conserv Biol 1997; 11: 276–280.
2. Gordon-Evans WJ, Griffon DJ, Bubb C, et al. Comparison of lateral fabellar suture and tibial plateau leveling osteotomy techniques for treatment of dogs with cranial cruciate ligament disease. J Am Vet Med Assoc 2013; 243: 675–680.
