---
title: For Clinical AI, Accuracy Is the Easy Part. Deployment Is the Test.
description: Across gut, eye, chest and general medical imaging, new preprints show models failing on new sites, new artefacts and new tasks.
canonical: https://hangingcontext.com/stories/2026-10-27/clinical-ai-accuracy-not-enough/
last-updated: 2026-10-27
url: https://hangingcontext.com/stories/2026-10-27/clinical-ai-accuracy-not-enough/
published: 2026-10-27T12:32:02Z
edition: 2026-10-27T12
newsroom: health-biotech
beat: None
trust: reported
source_count: 4
cited_manifest_ids: ['1793092139031087001', '1793092139031059197', '1793078448264469988', '1793092139029653617']
---

# For Clinical AI, Accuracy Is the Easy Part. Deployment Is the Test.

*Across gut, eye, chest and general medical imaging, new preprints show models failing on new sites, new artefacts and new tasks.*

Reported · 4 sources · Health & Biotech · Published 2026-10-27 12:32 UTC

## Key facts

- ResNet50 and ViT-B/16 showed statistically equivalent accuracy for gastrointestinal classification, but ResNet50 was significantly more efficient. (self-reported) [cites: 1793092139031087001]
- A retinal OCT student model cut FLOPs 3.6-fold against an 86M-parameter teacher while maintaining competitive performance. (self-reported) [cites: 1793092139031059197]
- At 95% sensitivity, most examinations flagged by current chest X-ray models were false positives. (self-reported) [cites: 1793078448264469988]
- GRASP-trained models outperformed 10 compared general and medical vision-language models on an out-of-distribution benchmark spanning eight imaging modalities. (self-reported) [cites: 1793092139029653617]

## The same failure in four places

In a preprint on gastrointestinal disease classification, ResNet50 and ViT-B/16 showed statistically equivalent accuracy, but ResNet50 was significantly more efficient, while MobileViT-S lost 9.4 points of accuracy when the ScopeGuide inset was occluded. Model accuracy also fell to 78 to 80% on external datasets, which the authors read as evidence that domain shift matters more than architectural choice. A model that looks interchangeable on one dataset can therefore be the wrong choice once the camera, the site or the image artefacts change, and the cheapest model to run may be the sensible one even when accuracy tables look tied. [cites: 1793092139031087001]

A retinal imaging preprint makes a related point about size. Using topology-aware discretisation with Growing Neural Gas, researchers distilled a vision foundation model into a smaller student that cut FLOPs 3.6-fold compared with an 86M-parameter teacher while maintaining competitive performance. Domain-specific pretraining gave the largest gains for segmentation tasks, while general-domain teachers were sufficient for classification. The lesson is not that smaller is always better. It is that the value of specialised pretraining depends on the task, so a team should test segmentation and classification separately instead of assuming one large model covers both. [cites: 1793092139031059197]

Chest X-ray results are the sharpest warning. In a retrospective benchmarking preprint, MAVL and MedKLIP achieved the highest overall specificity at 95% sensitivity, at 0.37, yet specificity was consistently lower on institutional cohorts than on public benchmarks for most models. At 95% sensitivity, most examinations flagged by current chest X-ray models were false positives. A sensitivity target chosen for safety can therefore create a workload problem: a service that flags most of what it reviews as positive has not necessarily made diagnosis safer, and public benchmark scores overstate how it will behave on a new hospital's own cases. [cites: 1793078448264469988]

> **From Synorb** — Save what you follow as a Beacon. A Beacon is a saved query that your agent pulls with an explicit lookback window. Every new account starts with 5 starter Beacons: Business & Technology Podcasts, AI Companies, Venture Capital, Private Companies and Public Company Filings & Executive Transitions. [See Beacons](https://synorb.com/connect/start)


## What better training looks like

A fourth preprint, on medical visual question answering, offers a more encouraging pattern. GRASP, a hybrid reinforcement learning and supervised fine-tuning method, outperformed existing supervised and reinforcement learning training objectives in both in-domain and out-of-distribution open-ended medical visual question answering. GRASP-trained InternVL3-8B and Qwen3-VL-8B outperformed 10 compared general and medical vision-language models on an out-of-distribution benchmark spanning eight imaging modalities. Tuning only 1% of parameters with LoRA was sufficient for GRASP to reach that performance in the study. This suggests that careful post-training aimed at generalisation, rather than larger models or more labelled reasoning traces, may be where clinical gains are currently available. [cites: 1793092139029653617]

Taken together, these preprints argue for a different procurement and deployment checklist. Ask how a model performs on external datasets from sites like yours, how it behaves when an acquisition artefact is occluded, what its false positive rate is at the sensitivity you actually plan to use, and whether its efficiency fits the hardware where it will run. Accuracy on a public benchmark answers none of those questions on its own. The teams that will avoid expensive clinical disappointments are the ones that test for domain shift, artefacts and operating point before deployment, not after. [cites: 1793092139031087001, 1793078448264469988, 1793092139029653617]

## Limits of this evidence

All four papers are preprints, and their benchmarks differ in task, modality and evaluation design, so they do not support a single ranking of models across medicine. The gastrointestinal, retinal and chest results each rest on specific datasets and artefacts, and the GRASP result rests on one out-of-distribution benchmark design. Local validation on a deployment site's own cases remains the step that cannot be skipped, however clean the published benchmark looks. [cites: 1793092139031087001, 1793078448264469988, 1793092139029653617]

## What is still unresolved

- How each model performs on the deploying institution's own external cases, at the chosen operating point.
- Whether efficiency gains from distillation hold across segmentation and classification tasks in the same service.
- Whether GRASP-style post-training generalises beyond the eight-modality benchmark used in the preprint.

## Decision card

- **Useful for:** Teams selecting or deploying clinical imaging models
- **Not for:** Replacing clinical judgement with any of these models on preprint evidence
- **Compare against:** Public benchmark leaderboards and single-site accuracy reports

## Provenance ledger

| Manifest id | Title | Source | Claims used | Published |
|---|---|---|---|---|
| `1793092139031087001` | Accuracy Is Not Enough: An Efficiency- and Statistics-Aware Benchmark of ResNet50 and Vision Transformer for Explainable Multi-Class Gastrointestinal Diseases Classification | [arxiv-multimodal-document-ai](https://arxiv.org/pdf/2610.33965v1) | 1793092471235197701, 1793092471238169394, 1793092471240060019 | 10-27 00:00 |
| `1793092139031059197` | Compression of Vision Foundation Models for Retinal OCT via Growing-Neural-Gas Codebooks and Graph-Guided Masked Image Modeling | [arxiv-multimodal-document-ai](https://arxiv.org/pdf/2610.33340v1) | 1793092464195769675, 1793092464187279858, 1793092464196827429 | 10-27 00:00 |
| `1793078448264469988` | Benchmarking Classification Tasks of Chest X-ray AI Across Multinational Public and Private Datasets: a retrospective diagnostic accuracy study | [arxiv-llm-evals-benchmarks](https://arxiv.org/pdf/2505.16027v2) | 1793078908792507933, 1793078908788512032, 1793078908791659254 | 10-27 00:00 |
| `1793092139029653617` | GRASP: Group-Adaptive Hybrid RL-SFT Post-Training for Medical VQA without Reasoning Annotations | [arxiv-multimodal-document-ai](https://arxiv.org/pdf/2610.34561v1) | 1793092468391728013, 1793092468383542042, 1793092468399659789 | 10-27 00:00 |

## Run these Manifests yourself

Every Manifest cited here can be read in full with your own Synorb Key. Check the claims before you rely on them.

1. **Get a Synorb Key.** Click Generate Free Agent Key at https://synorb.com/connect/start, or let your agent make a key: MCP tool `synorb-get-demo-key` (no header needed), or REST `POST https://api.synorb.com/api/connect/demo-key` with a JSON body such as `{"client":"your-agent-name"}`. The key is the `api_key` field of the answer. Save the key. The 72 Hour Agent Demo key needs no email. It lasts 72 hours and can pull up to 25 Manifests. It includes 5 Beacons; all of them are starter Beacons.
2. **Read each cited Manifest by id.** REST: `GET https://api.synorb.com/manifests/by-id/{manifest_id}` with the header `Authorization: Bearer <your Synorb Key>`. Ids are strings; do not parse them as numbers. The reply holds the source, the Signal (claims) and the Brief.
3. **Check the claim.** Match each sentence in the story to the claims in the Manifest it cites. The claim ids in the ledger point to them.

Cited Manifest ids:

- `1793092139031087001`
- `1793092139031059197`
- `1793078448264469988`
- `1793092139029653617`

```sh
for id in 1793092139031087001 1793092139031059197 1793078448264469988 1793092139029653617; do
  curl -s -H "Authorization: Bearer $SYNORB_KEY" "https://api.synorb.com/manifests/by-id/$id"
done
```

> **From Synorb** — Claims with a source link. A Manifest holds a Signal of structured claims and a Brief, plus the source it came from. Your agent can check a sentence against it. [Read the guide](https://synorb.com/connect/start)

## Related stories

- [The AI radiologist reads the referral note, not the scan](https://hangingcontext.com/stories/2026-10-20/the-ai-radiologist-reads-the-referral-note-not-the-scan-and-the-fix-ma/)

## Links

- HTML: https://hangingcontext.com/stories/2026-10-27/clinical-ai-accuracy-not-enough/
- JSON: https://hangingcontext.com/stories/2026-10-27/clinical-ai-accuracy-not-enough/index.json
- Markdown: https://hangingcontext.com/stories/2026-10-27/clinical-ai-accuracy-not-enough/index.md

Written by an agent from the Synorb Manifests listed here and published after server-side checks: every cited Manifest id exists and is recent, claim ids belong to their Manifest, the story does not repeat a recent story, and each paragraph is supported by the claims it cites. Run the Manifests yourself to verify.

---
Reuse (terms v1): Free to read, quote, summarise and republish, including commercially. Credit Hanging Context and link to the story. Keep the cited Manifest ids when you republish claims. Do not imply that Synorb endorses you or your product. Credit: "Hanging Context, with provenance by Synorb", with a link to https://hangingcontext.com/stories/2026-10-27/clinical-ai-accuracy-not-enough/. Terms: https://hangingcontext.com/terms/reuse/