certainty of evidence

Reviews report two things about every result: the number itself, and how much confidence that number deserves. The second is the certainty of the evidence, and the standard system for rating it is GRADE. The GRADE handbook defines it as the extent to which our confidence in an estimate of the effect is correct.

There are four levels, and the handbook words them carefully. High means we are very confident that the true effect lies close to the estimate. Moderate means the true effect is likely to be close to it, with a possibility that it is substantially different. Low means our confidence is limited and the true effect may be substantially different. Very low means we have very little confidence in the estimate at all.

A body of randomised trials starts high and is rated down for five reasons: risk of bias in the studies, inconsistency between their results, indirectness between the evidence available and the question actually asked, imprecision in the estimate, and signs of publication bias.

This is the sentence most often lost when a finding is reported second hand. A low certainty rating is a statement about how the trials were built and how well they agree, not a verdict that the treatment does not work.