publication bias

Cochrane's methods manual, the standard reference for how evidence is pooled, defines the problem as non-reporting bias: it arises when decisions about how, when or where the results of eligible studies are reported depend on the P value, the size or the direction of those results. Publication bias and reporting bias are the older names for the same thing, and the handbook now prefers non-reporting bias because the results that go missing are not only the ones nobody published at all. The pattern it describes is well documented. Results that are statistically significant, and that suggest an intervention works, are more likely to be available, more likely to be available quickly, more likely to appear in high impact journals and more likely to be cited by others.

This is a different question from whether any single study was well run. Risk of bias asks whether one study's own result can be trusted. Non-reporting bias asks whether the set of results a review managed to find is the whole set. A meta-analysis can only pool what it can locate, so if what is missing is missing because of what it showed, the pooled figure is shifted in a predictable direction rather than blurred at random. Where a field is moving quickly, delay alone can produce it: studies with positive results become available sooner than the others, which the handbook calls time-lag bias, so a review run today may be reading a literature that will look different in two years.

The usual check is a funnel plot, which the handbook describes as a scatter plot of the effect estimates from individual studies against a measure of each study's size or precision. If the small studies scatter evenly around the same answer as the large ones, the plot looks symmetrical. Asymmetry is where suspicion starts, and the handbook is firm that it is not diagnostic, because the causes include real bias in the study results, genuine heterogeneity between the studies, and artefacts of the statistics themselves. Sitting alongside it are small-study effects, the tendency for the effects estimated in smaller studies to differ from those estimated in larger ones, which is what the statistical tests in this area actually detect. Those tests, of which Egger's regression is the best known, typically have low power, do not always reach the same conclusion as one another, and are recommended only when at least 10 studies are available and those studies vary in size. Reading a funnel plot by eye is subjective as well. A review reporting no sign of publication bias is therefore reporting that a weak test did not fire, which is not the same thing as the evidence being complete.