heterogeneity

When a review pools several studies, their results never line up exactly. Some of the scatter is ordinary sampling error, the noise expected from measuring different groups of people. Heterogeneity is what is left over. The Cochrane Handbook defines it as variability in the intervention effects being evaluated in the different studies, and it shows up when the observed effects differ by more than random chance would produce.

Reviews usually quantify it with a statistic called I squared, which describes the percentage of the variability in effect estimates that is due to heterogeneity rather than to sampling error. Cochrane offers rough bands for reading it: nought to 40 percent might not be important, 30 to 60 percent may be moderate, 50 to 90 percent substantial, and 75 to 100 percent considerable. The bands overlap deliberately, because the number matters less than the size and direction of the effects behind it.

High heterogeneity does not invalidate a review. It changes what the pooled number is good for. A summary drawn from studies that disagree sharply is an honest description of the research literature and a poor prediction of what any one person would experience.