statistical power

Power is a property of the study rather than of its result, fixed before any data arrive. It depends mostly on how many participants there are and how large a difference is being looked for: a small study hunting a small difference has a poor chance of finding it even when the difference is real. Writing in the BMJ in 1995, the medical statisticians Douglas Altman and Martin Bland gave an example. A published trial comparing two treatments for variceal bleeding enrolled 100 patients despite its own calculation putting the requirement at 1,800, and so had about a 5 percent chance of reaching statistical significance if the treatment difference the authors had specified truly existed.

The consequence is the misreading that gives their note its title, absence of evidence is not evidence of absence. Altman and Bland argue that calling a trial that found no statistically detectable difference a "negative" trial wrongly implies it showed there is no difference, when usually all it has shown is an absence of evidence of one. Those are different statements. They cite a survey of trials published in the New England Journal of Medicine which found that only 30 percent of those reporting a P value above 0.1 were large enough to have a 90 percent chance of detecting even a 50 percent difference in the effectiveness of the treatments compared.

Their practical test is to look past the P value for a quantification of the effect and the width of its confidence interval. In their example the trial's authors concluded that the two treatments were equally effective while their own 95 percent confidence interval still allowed a difference in cure rates of up to 20 percentage points. A wide interval sitting around a finding of no difference is the signature of a study that lacked the power to answer the question it asked, and it is the reason a small trial's null result settles much less than a large one's.