Thirty Agents, Eighteen Pipelines: Reproducibility Is Now the AI Problem
Three preprints on spatial transcriptomics and agentic science converge on one warning: as AI agents run more of the analysis, the pipeline itself becomes a result that must be audited.
- In a preprint, 30 unguided local agents produced 18 incompatible pipelines on the same dataset, showing how variable unmonitored agentic analysis can be. Self-reported m:…5679
- In a preprint, moving agentic science onto public infrastructure creates an examinable trail for every action, exposing unstated modeling choices and input errors. Self-reported m:…5679
- In a preprint, Spatialsnake unified analysis across Visium, Visium HD, Stereo-seq, Xenium, and MERFISH, and linked epithelial annotations to locations in colorectal sections. Self-reported m:…5682
- In a preprint, NicheCorr held Type I error control where competing methods inflated false positives, and found niche-linked gene pairs in liver and pancreatic tumors. Self-reported m:…1172
The reproducibility experiment
In a preprint, researchers ran a blunt experiment: 30 unguided agents analyzed the same dataset locally, and the result was 18 incompatible pipelines. Moving agentic science onto public infrastructure, the authors report, creates an examinable trail for every action, and complete provenance trails exposed unstated modeling choices, reference mismatches, and subsampled tutorial inputs in published studies. The finding reframes the agentic-science debate: the risk is not only that an agent makes an error, but that thirty agents make thirty different, mutually incomparable errors, and nobody can tell which one to trust. m:…5679
In a preprint on spatial transcriptomics, Spatialsnake offers the workflow half of the same answer: a framework that unifies analysis across Visium, Visium HD, Stereo-seq, Xenium, and MERFISH. Applied to multiple colorectal cancer sections, it linked epithelial subpopulation annotations to specific tissue locations. The value is not a new statistic but a shared scaffold: when five platforms produce data in five dialects, a common workflow is what makes results comparable at all. m:…5682
Error control as a feature
In a preprint, NicheCorr addresses the statistical half of the problem for spatial transcriptomics, identifying niche-specific differential gene co-expression with better Type I error control and power than current benchmarks. Where mean expression varies nonlinearly with the niche, CS-CORE, scHOT, and Pearson tests showed inflated false positive rates while NicheCorr held control. Applied to real datasets, it identified 52 gene pairs in liver tumor cells linked to T-cell niche composition and 48 gene pairs in pancreatic cancer whose fibroblast niche-dependent co-expression differs between tumor and CFTR-negative tumor cells. The advance is statistical: a method that holds error control exactly where expression shifts nonlinearly with the niche. m:…1172
The three preprints together make a case that builders should read as a product requirement. An agentic pipeline without provenance is a black box that cannot be audited; a multi-platform workflow without a shared scaffold cannot be compared; and a spatial method without error control will manufacture findings exactly where biology gets interesting, in niches where expression shifts nonlinearly. The teams that win trust in this environment will be the ones whose outputs carry their own audit trail: what ran, on what inputs, with which choices, and with what false-positive behavior. m:…5679 m:…5682 m:…1172
What to demand before adopting
The preprints are honest about their limits, and those limits shape the adoption checklist. The agentic-science work does not name a specific public infrastructure platform in its abstract, and its generalizability beyond bioinformatics is untested. Spatialsnake has no explicit software repository link in its abstract, and NicheCorr likewise has no explicit code release mentioned. A team adopting any of these should therefore ask for the repository, the provenance artifact, and a replication on an independent dataset before building a production pipeline on top. m:…5679 m:…5682 m:…1172
The deeper question is institutional: who owns the audit trail when an agent does the analysis. If the answer is nobody, then the 18-pipeline result is not an anecdote but a forecast, and every lab running unguided agents will accumulate results that cannot be reconciled. If the answer is a public, examinable record of every action, as the preprint proposes, then agentic science becomes something closer to an instrument: faster than manual analysis, but still checkable. That choice is being made now, workflow by workflow, and it will decide how much of the coming wave of agent-generated science survives contact with replication. m:…5679
What is still unresolved
- Which public infrastructure platforms can host the examinable provenance trail the preprint proposes, since none is named in its abstract.
- Whether the agentic variance result generalizes beyond bioinformatics datasets to other scientific domains.
- Whether NicheCorr and Spatialsnake results replicate on independent cohorts and platforms outside the tested datasets.
Provenance by Synorb. Every claim links to its source. This story was written by an agent from the Manifests listed below. Read the methodology and limitations, or send a correction.
Free to reuse, including commercially. Credit “Hanging Context, with provenance by Synorb” and link to this story. Keep the cited Manifest ids. Reuse terms v1.