Biology's benchmarks are measuring the wrong thing, four new preprints argue

The averaging problem

The first problem is what a benchmark throws away before scoring starts. In a preprint on gene perturbation benchmarks, averaging cells obscures perturbation-strength variation, and higher perturbation strength is associated with greater transcriptional response reproducibility. That is an awkward fact for any leaderboard built on averaged cells: the cells that were barely perturbed and the cells that were strongly perturbed do not behave the same way, yet the average treats them as one measurement. A model scored on that average is being graded on a blend, not on a response. m:…0441

The practical consequence shows up when models are asked to identify what was done to a cell. In the same preprint, STATE and Cell2Sentence models do not reliably recover perturbation identity in HCT116 retrieval tests. If a model cannot reliably say which perturbation produced a profile, then claims about that model predicting perturbation effects rest on shaky ground. The implication is that builders should check identity recovery before celebrating effect prediction, because a model that cannot tell perturbations apart is unlikely to be reasoning about their biology. m:…0441

The endpoint problem

The second preprint makes a related point about what the benchmark asks for. Its InverseBench framework allows for the explicit construction of candidate information and endpoints in perturbation benchmarks, so the choice of endpoint is out in the open rather than buried in the evaluation code. That matters because the FSTN pipeline's performance is highly sensitive to the choice of endpoint, showing a significant gap between transcriptional proxy accuracy and growth fitness prediction. A model can look strong on a transcriptional proxy and much weaker on an independent biological outcome such as growth fitness, and both numbers can be honestly reported from the same pipeline. m:…5854

The authors add a caution that cuts against the simplest reading of that gap: endpoint-dependent accuracy gaps do not by themselves mean worse biological decisions. A proxy and a fitness outcome are different questions, and a model can be useful for one without being good at the other. The lesson for anyone reading a benchmark is to ask which endpoint the headline number belongs to, and whether that endpoint is the decision the reader actually has to make. m:…5854

The identity problem

The third preprint shows the same trap in clinical omics, where the confounder is not an endpoint but a label. Genotype-based prediction for systemic lupus erythematosus using 30 candidate SNPs is largely dependent on SNPs that deviate from Hardy-Weinberg equilibrium in controls, which means the genetic signal leans on variants that already look unusual in the control group. Multi-study gene expression models for the same disease are highly confounded by study identity, which can achieve an ROC-AUC of 0.857 on its own. In plain terms, a model can appear to detect disease when it is partly detecting which study a sample came from. m:…0212

The fix the authors point to is unglamorous and necessary: leakage-free cross-validation is required to accurately estimate model performance in multi-study omics, because standard cross-validation fails to account for batch-label confounding. On the positive side, lower MTR expression is the most consistent expression feature for the disease across the analyzed multi-study datasets, so there is a real signal underneath the confounding. The distinction matters for builders: a model that survives leakage-free evaluation is worth attention, and one that has only passed standard cross-validation has not yet been tested against its most likely shortcut. m:…0212

What good measurement looks like

The fourth preprint is the most constructive of the set, because it shows what a better test looks like rather than only criticizing existing ones. It benchmarks 12 differential rhythmicity methods from 5 packages using synthetic RNA-seq data, and finds that synthetic models incorporating empirically estimated mean-dispersion trends provide a more accurate benchmark for differential rhythmicity analysis than standard synthetic data. In other words, the realism of the synthetic data is itself a measurement choice, and getting it closer to empirical patterns changes which methods look best. m:…6320

Read together, the four preprints point to one discipline: report the construction of the test alongside the score. Report how perturbation strength was distributed, not just the average response. Report the endpoint, and whether a proxy stands in for an outcome the reader cares about. Report a leave-one-study-out style check before claiming a disease signal. None of that is as satisfying as a single leaderboard number, but the evidence here suggests the single number is exactly where the misleading part lives. For teams building biology models, the cheapest improvement available is to stop trusting a benchmark that has not shown its own seams. m:…0441 m:…5854 m:…0212 m:…6320