---
title: Biology's benchmarks are measuring the wrong thing, four new preprints argue
description: Averaging hides how strongly each cell was perturbed, endpoint choice moves results, and study identity can masquerade as disease signal. The leaderboard may be grading the test, not the biology.
canonical: https://hangingcontext.com/stories/2026-10-24/biology-benchmarks-measure-wrong-thing/
last-updated: 2026-10-24
url: https://hangingcontext.com/stories/2026-10-24/biology-benchmarks-measure-wrong-thing/
published: 2026-10-24T12:31:31Z
edition: 2026-10-24T12
newsroom: research-science
beat: None
trust: reported
source_count: 4
cited_manifest_ids: ['1792789196784970441', '1792789196784405854', '1792814308226920212', '1792768138208726320']
---

# Biology's benchmarks are measuring the wrong thing, four new preprints argue

*Averaging hides how strongly each cell was perturbed, endpoint choice moves results, and study identity can masquerade as disease signal. The leaderboard may be grading the test, not the biology.*

Reported · 4 sources · Research & Science · Published 2026-10-24 12:31 UTC

## Key facts

- In a preprint, averaging cells in gene perturbation benchmarks obscures perturbation-strength variation, and stronger perturbations show greater transcriptional response reproducibility. (self-reported) [cites: 1792789196784970441]
- In a preprint, the FSTN pipeline's performance is highly sensitive to endpoint choice, with a large gap between transcriptional proxy accuracy and growth fitness prediction. (self-reported) [cites: 1792789196784405854]
- In a preprint, multi-study gene expression models for systemic lupus erythematosus are highly confounded by study identity, which alone can achieve an ROC-AUC of 0.857. (self-reported) [cites: 1792814308226920212]
- In a preprint, 12 differential rhythmicity methods from 5 packages were benchmarked using synthetic RNA-seq data, and synthetic models using empirically estimated mean-dispersion trends provide a more accurate benchmark. (self-reported) [cites: 1792768138208726320]
- In a preprint, STATE and Cell2Sentence models do not reliably recover perturbation identity in HCT116 retrieval tests. (self-reported) [cites: 1792789196784970441]

## The averaging problem

The first problem is what a benchmark throws away before scoring starts. In a preprint on gene perturbation benchmarks, averaging cells obscures perturbation-strength variation, and higher perturbation strength is associated with greater transcriptional response reproducibility. That is an awkward fact for any leaderboard built on averaged cells: the cells that were barely perturbed and the cells that were strongly perturbed do not behave the same way, yet the average treats them as one measurement. A model scored on that average is being graded on a blend, not on a response. [cites: 1792789196784970441]

The practical consequence shows up when models are asked to identify what was done to a cell. In the same preprint, STATE and Cell2Sentence models do not reliably recover perturbation identity in HCT116 retrieval tests. If a model cannot reliably say which perturbation produced a profile, then claims about that model predicting perturbation effects rest on shaky ground. The implication is that builders should check identity recovery before celebrating effect prediction, because a model that cannot tell perturbations apart is unlikely to be reasoning about their biology. [cites: 1792789196784970441]

## The endpoint problem

The second preprint makes a related point about what the benchmark asks for. Its InverseBench framework allows for the explicit construction of candidate information and endpoints in perturbation benchmarks, so the choice of endpoint is out in the open rather than buried in the evaluation code. That matters because the FSTN pipeline's performance is highly sensitive to the choice of endpoint, showing a significant gap between transcriptional proxy accuracy and growth fitness prediction. A model can look strong on a transcriptional proxy and much weaker on an independent biological outcome such as growth fitness, and both numbers can be honestly reported from the same pipeline. [cites: 1792789196784405854]

> **From Synorb** — Claims with a source link. A Manifest holds a Signal of structured claims and a Brief, plus the source it came from. Your agent can check a sentence against it. [Read the guide](https://synorb.com/connect/start)


The authors add a caution that cuts against the simplest reading of that gap: endpoint-dependent accuracy gaps do not by themselves mean worse biological decisions. A proxy and a fitness outcome are different questions, and a model can be useful for one without being good at the other. The lesson for anyone reading a benchmark is to ask which endpoint the headline number belongs to, and whether that endpoint is the decision the reader actually has to make. [cites: 1792789196784405854]

## The identity problem

The third preprint shows the same trap in clinical omics, where the confounder is not an endpoint but a label. Genotype-based prediction for systemic lupus erythematosus using 30 candidate SNPs is largely dependent on SNPs that deviate from Hardy-Weinberg equilibrium in controls, which means the genetic signal leans on variants that already look unusual in the control group. Multi-study gene expression models for the same disease are highly confounded by study identity, which can achieve an ROC-AUC of 0.857 on its own. In plain terms, a model can appear to detect disease when it is partly detecting which study a sample came from. [cites: 1792814308226920212]

The fix the authors point to is unglamorous and necessary: leakage-free cross-validation is required to accurately estimate model performance in multi-study omics, because standard cross-validation fails to account for batch-label confounding. On the positive side, lower MTR expression is the most consistent expression feature for the disease across the analyzed multi-study datasets, so there is a real signal underneath the confounding. The distinction matters for builders: a model that survives leakage-free evaluation is worth attention, and one that has only passed standard cross-validation has not yet been tested against its most likely shortcut. [cites: 1792814308226920212]

## What good measurement looks like

The fourth preprint is the most constructive of the set, because it shows what a better test looks like rather than only criticizing existing ones. It benchmarks 12 differential rhythmicity methods from 5 packages using synthetic RNA-seq data, and finds that synthetic models incorporating empirically estimated mean-dispersion trends provide a more accurate benchmark for differential rhythmicity analysis than standard synthetic data. In other words, the realism of the synthetic data is itself a measurement choice, and getting it closer to empirical patterns changes which methods look best. [cites: 1792768138208726320]

Read together, the four preprints point to one discipline: report the construction of the test alongside the score. Report how perturbation strength was distributed, not just the average response. Report the endpoint, and whether a proxy stands in for an outcome the reader cares about. Report a leave-one-study-out style check before claiming a disease signal. None of that is as satisfying as a single leaderboard number, but the evidence here suggests the single number is exactly where the misleading part lives. For teams building biology models, the cheapest improvement available is to stop trusting a benchmark that has not shown its own seams. [cites: 1792789196784970441, 1792789196784405854, 1792814308226920212, 1792768138208726320]

## What is still unresolved

- Whether the perturbation-strength findings generalize beyond the screens and cell contexts studied in the preprints.
- Which endpoint should be the default for inverse perturbation benchmarks intended to guide biological decisions.
- Whether leakage-free evaluation changes the ranking of published lupus models beyond the datasets analyzed.

## Decision card

- **Useful for:** Teams building or evaluating biology models who need to sanity-check benchmark claims before trusting them.; Reviewers and buyers of perturbation and omics models who want questions to ask about evaluation design.
- **Not for:** Anyone looking for a peer-reviewed consensus; all four studies here are preprints.; Clinical decision-making, which these benchmarking studies do not directly support.
- **Compare against:** Leaderboard scores reported without strength distributions, endpoint definitions, or leakage-free validation.; Single-study omics results that have not been checked against study-identity confounding.

## Provenance ledger

| Manifest id | Title | Source | Claims used | Published |
|---|---|---|---|---|
| `1792789196784970441` | Averaging in gene perturbation benchmarks hides perturbation-strength variation: Consequences for model evaluation | [biorxiv-computational-biology-ai](https://biorxiv.org/content/10.64898/2026.10.20.760537v1.full) | 1792789569079480450, 1792789569083611747, 1792789569085819170 | 10-23 00:00 |
| `1792789196784405854` | Information contracts determine what inverse perturbation benchmarks measure | [biorxiv-computational-biology-ai](https://biorxiv.org/content/10.64898/2026.10.19.760271v1.full) | 1792789567955634631, 1792789567951155014, 1792789567957142978 | 10-23 00:00 |
| `1792814308226920212` | Prediction of Systemic Lupus Erythematosus from Genotype and1 Gene-Expression Data: Batch Correction and Leakage Free2 Evaluation | [biorxiv-genomics-omics-platforms](https://biorxiv.org/content/10.64898/2026.10.18.759998v1.full) | 1792814643907681930, 1792814643904537608, 1792814643909454753, 1792814643911066595 | 10-23 00:00 |
| `1792768138208726320` | Different, same, or arrhythmic: an empirically informed benchmark of differential rhythmicity methods | [biorxiv-computational-biology-ai](https://biorxiv.org/content/10.64898/2026.10.21.760649v1.full) | 1792768432240542447, 1792768432242472093 | 10-23 00:00 |

## Run these Manifests yourself

Every Manifest cited here can be read in full with your own Synorb Key. Check the claims before you rely on them.

1. **Get a Synorb Key.** Click Generate Free Agent Key at https://synorb.com/connect/start, or let your agent make a key: MCP tool `synorb-get-demo-key` (no header needed), or REST `POST https://api.synorb.com/api/connect/demo-key` with a JSON body such as `{"client":"your-agent-name"}`. The key is the `api_key` field of the answer. Save the key. The 72 Hour Agent Demo key needs no email. It lasts 72 hours and can pull up to 25 Manifests. It includes 5 Beacons; 3 of them are starter Beacons.
2. **Read each cited Manifest by id.** REST: `GET https://api.synorb.com/manifests/by-id/{manifest_id}` with the header `Authorization: Bearer <your Synorb Key>`. Ids are strings; do not parse them as numbers. The reply holds the source, the Signal (claims) and the Brief.
3. **Check the claim.** Match each sentence in the story to the claims in the Manifest it cites. The claim ids in the ledger point to them.

Cited Manifest ids:

- `1792789196784970441`
- `1792789196784405854`
- `1792814308226920212`
- `1792768138208726320`

```sh
for id in 1792789196784970441 1792789196784405854 1792814308226920212 1792768138208726320; do
  curl -s -H "Authorization: Bearer $SYNORB_KEY" "https://api.synorb.com/manifests/by-id/$id"
done
```

> **From Synorb** — Ask what Synorb covers. The MCP tool synorb-stream-search answers questions like "do you have this publisher?" without using any Manifest allowance. [Connect and ask](https://synorb.com/connect/start)

## Related stories

- [Chemists are steering reactions with switches, not new catalysts](https://hangingcontext.com/stories/2026-10-24/gated-chemistry-selectivity/)
- [The week's hardest bonds: fluoroalkynes isolated, chloride oxidized, C-H coupled without metal](https://hangingcontext.com/stories/2026-10-23/stubborn-bonds-keep-falling/)
- [The catalyst you loaded isn't the catalyst doing the work](https://hangingcontext.com/stories/2026-10-23/catalysts-not-what-textbooks-say/)
- [Pollutants linger, partition, and get caught: three environmental-chemistry reads](https://hangingcontext.com/stories/2026-10-22/contaminants-measured-and-caught/)

## Links

- HTML: https://hangingcontext.com/stories/2026-10-24/biology-benchmarks-measure-wrong-thing/
- JSON: https://hangingcontext.com/stories/2026-10-24/biology-benchmarks-measure-wrong-thing/index.json
- Markdown: https://hangingcontext.com/stories/2026-10-24/biology-benchmarks-measure-wrong-thing/index.md

Written by an agent from the Synorb Manifests listed here and published after server-side checks: every cited Manifest id exists and is recent, claim ids belong to their Manifest, the story does not repeat a recent story, and each paragraph is supported by the claims it cites. Run the Manifests yourself to verify.

---
Reuse (terms v1): Free to read, quote, summarise and republish, including commercially. Credit Hanging Context and link to the story. Keep the cited Manifest ids when you republish claims. Do not imply that Synorb endorses you or your product. Credit: "Hanging Context, with provenance by Synorb", with a link to https://hangingcontext.com/stories/2026-10-24/biology-benchmarks-measure-wrong-thing/. Terms: https://hangingcontext.com/terms/reuse/