Imprecise tests risk false safety claims in infant TB vaccine trials

Vaccines to protect newborns and infants against tuberculosis are being tested in trials that compare new candidates against the established Bacille Calmette-Guérin (BCG) vaccine. For these trials, researchers are increasingly using evidence of tuberculosis infection — rather than disease — as the main outcome, and they are doing so in a trial format called a non-inferiority design. In a non-inferiority trial the goal is to show that a new vaccine does not perform substantially worse than the standard vaccine. But there is a key problem: the laboratory markers used to detect tuberculosis infection do not work perfectly. They have less-than-perfect sensitivity (they miss some infections) and specificity (they can give false positives). Daniel Grint and colleagues explored how these imperfections interact with the logic of non-inferiority trials. They note that common flaws in non-inferiority designs tend to push results toward no difference between groups — the so-called null — which increases the risk that a new vaccine will be declared ‘non-inferior’ simply because the tests are imprecise, not because the vaccine truly performs as well as BCG.

To quantify this risk, the team ran a statistical simulation study. They generated trial data under three different 2-year cumulative risks of tuberculosis infection — 2%, 5% and 8% — and varied the quality of the infection tests. Test specificity was modeled from perfect (100%) down to 85%, and sensitivity from 100% down to 64%. The analysis used log-binomial regression to estimate relative risk of infection between the two randomized arms. When sensitivity and specificity were both perfect at 100%, the observed type I error (false positive rate) and type II error (power) matched expected values: about 2.5% and 80% respectively across all three infection-risk scenarios. However, even modest drops in test quality had large effects. With sensitivity and specificity both at 95%, the simulated risk of falsely declaring non-inferiority was 96.8% in the 2% risk scenario, 53.2% in the 5% scenario, and 27.8% in the 8% scenario. These results show that small departures from perfect test accuracy can dramatically increase the chance of a wrong conclusion.

The implications for vaccine research are serious and immediate: trials that use infection as the primary outcome must take into account how well the infection tests actually perform. The simulations by Daniel Grint and colleagues show that failing to account for imperfect specificity in particular can lead to a default finding of non-inferiority — not because the new vaccine is as good as BCG, but because the measurement tools blur real differences. Practically, that means trial designers and regulators should plan for the limits of diagnostic markers when choosing endpoints, setting non-inferiority margins, and calculating sample sizes. It also suggests that where infection markers are imperfect, trials may need different designs, alternative endpoints, or stronger statistical adjustment to avoid approving vaccines on the basis of misleading results. In short, the precision of the tools used to define infection matters as much as the vaccines themselves: ignoring that can let inferior vaccines pass as acceptable.