risk of bias

Bias here does not mean carelessness or bad faith. It means a systematic distortion: a feature of how a study was built or run that could shift its result in a particular direction, however carefully the numbers were then calculated. Assessing risk of bias asks how vulnerable a trial is to that, which is a different question from how large it was or how clearly it was written up.

Cochrane's current tool for randomised trials, RoB 2, works through five domains: bias arising from the randomisation process, bias due to deviations from the intended interventions, bias due to missing outcome data, bias in measurement of the outcome, and bias in selection of the reported result.

Two familiar safeguards sit inside those domains. Allocation concealment hides the next assignment from whoever enrols each participant, and Cochrane notes it can always be implemented whatever the design. Blinded outcome assessment keeps whoever measures the result unaware of which group a person was in, and that one cannot always be implemented, which is a structural difficulty in trials of exercise, surgery and talking therapies rather than a failing of any single study.

Reviews often repeat an analysis using only the trials at low risk of bias. Where the effect shrinks, part of the original estimate was probably coming from how the weaker studies were run.