BIG-TB: testing AI on tuberculosis genomes

Predicting which strains of Mycobacterium tuberculosis resist antibiotics is a critical goal for public health, and many researchers hope machine learning can help. Anna G. Green and colleagues created a new resource called Benchmarks for Interpretable prediction from Genomes of Tuberculosis, BIG-TB, to test how well sequence-focused artificial intelligence methods actually work. The team gathered over 17,000 genomes with high-quality short read sequencing data and experimentally measured antibiotic resistance phenotypes, and paired those genomes with a curated list of canonical resistance-conferring variants. All of this was packaged in an ML-ready format so models can be trained and evaluated consistently. BIG-TB is designed to do more than score raw prediction accuracy: it explicitly tests interpretability by asking whether a model’s predictions can be traced back to known, expert-curated resistance variants. By bringing together a large, standardized dataset and a clear set of tasks, Anna G. Green and the group aim to move evaluation of biological foundation models beyond a small set of benchmarks and toward practical use cases in tuberculosis research.

BIG-TB defines two main tasks for testing models: (1) predicting antibiotic resistance phenotypes from sequence data, and (2) attributing those predictions to known resistance-conferring variants from an expert-curated dataset. Using this benchmark, the authors compared DNA-based foundation models, protein-based models, and simpler machine learning baselines. They report that DNA-based foundation models do not yet outperform simple machine learning baselines, with mean test AUC = 0.888 vs. 0.846 for best CNN variant vs. best DNABERT variant across drugs. Protein-based models performed worse than DNA-based models because they lose representation of non-coding variants, although foundation models were more competitive in the protein space. The benchmark also revealed that models with the highest predictive performance do not necessarily perform best at canonical resistance variant discovery, suggesting some gains come from non-causal associations. Finally, the authors found that the choice of embedding representation strongly affects foundation model performance, and that representations that average over sequence position perform poorly at both prediction and canonical resistance variant discovery. Code is available at https://github.com/SAGE-Lab-UMass/Big-TB-benchmark.

BIG-TB offers a new way to evaluate sequence-based foundation models that focuses both on prediction and interpretability. By including an expert-curated list of canonical resistance variants and a large set of genomes with measured phenotypes, the benchmark makes it possible to compare how well models predict resistance and whether they base those predictions on biologically meaningful signals. The finding that simple models can still outperform some foundation models, and that better predictive scores do not always mean better discovery of causal variants, is a caution for researchers and clinicians who might assume bigger models are always better. The sensitivity of results to embedding choices highlights a practical lever for improving models: better representations that preserve positional and non-coding information may be crucial. Overall, BIG-TB should help developers test new models more rigorously, prioritize interpretability, and focus improvements where they will most likely lead to reliable, clinically useful tools.