sensitivity analysis

Analysing a study involves choices that could honestly have gone another way: who to count, what to do about the people who dropped out, which measurement to treat as the outcome. A sensitivity analysis is the main analysis run again under one of those alternative choices, to see whether the answer moves. ICH E9, the guideline on statistical principles for clinical trials agreed by the regulators of Europe, the United States and Japan, treats this as a standard obligation rather than an optional extra: in its section on summarising a clinical database it says the sensitivity of conclusions to the assumptions and selections made should be explored.

Two occasions come up most often, and the guideline addresses both. The first is missing data. Where participants leave a trial before the end, ICH E9 says an investigation should be made into how sensitive the results are to the method of handling the missing values, and that this matters most when the number of them is substantial. The second is the choice of who gets analysed. A trial can be analysed on everyone randomised, or on the narrower group who completed the protocol as written, and the guideline calls it advantageous to demonstrate what it terms "a lack of sensitivity of the principal trial results" to that choice. When the two sets lead to essentially the same conclusions, it says, confidence in the trial results is increased.

The reading follows from that. A sensitivity analysis that agrees with the main analysis does not add a new result, it removes a doubt: the finding was not an artefact of one defensible decision. A sensitivity analysis that disagrees is the more informative outcome, because it locates the result's dependence on a particular choice, and a paper that reports one is showing its working rather than undermining itself. Two things it is not: it is not a subgroup analysis, which asks whether an effect differs between kinds of participant rather than whether it survives a different analytic choice, and it is not a search for a better answer among several attempts. ICH E9 asks for these analyses to be planned in the protocol for that reason, so that the alternative was named before anyone knew what it would produce.