For Clinical AI, Accuracy Is the Easy Part. Deployment Is the Test.

The same failure in four places

In a preprint on gastrointestinal disease classification, ResNet50 and ViT-B/16 showed statistically equivalent accuracy, but ResNet50 was significantly more efficient, while MobileViT-S lost 9.4 points of accuracy when the ScopeGuide inset was occluded. Model accuracy also fell to 78 to 80% on external datasets, which the authors read as evidence that domain shift matters more than architectural choice. A model that looks interchangeable on one dataset can therefore be the wrong choice once the camera, the site or the image artefacts change, and the cheapest model to run may be the sensible one even when accuracy tables look tied. m:…7001

A retinal imaging preprint makes a related point about size. Using topology-aware discretisation with Growing Neural Gas, researchers distilled a vision foundation model into a smaller student that cut FLOPs 3.6-fold compared with an 86M-parameter teacher while maintaining competitive performance. Domain-specific pretraining gave the largest gains for segmentation tasks, while general-domain teachers were sufficient for classification. The lesson is not that smaller is always better. It is that the value of specialised pretraining depends on the task, so a team should test segmentation and classification separately instead of assuming one large model covers both. m:…9197

Chest X-ray results are the sharpest warning. In a retrospective benchmarking preprint, MAVL and MedKLIP achieved the highest overall specificity at 95% sensitivity, at 0.37, yet specificity was consistently lower on institutional cohorts than on public benchmarks for most models. At 95% sensitivity, most examinations flagged by current chest X-ray models were false positives. A sensitivity target chosen for safety can therefore create a workload problem: a service that flags most of what it reviews as positive has not necessarily made diagnosis safer, and public benchmark scores overstate how it will behave on a new hospital's own cases. m:…9988

What better training looks like

A fourth preprint, on medical visual question answering, offers a more encouraging pattern. GRASP, a hybrid reinforcement learning and supervised fine-tuning method, outperformed existing supervised and reinforcement learning training objectives in both in-domain and out-of-distribution open-ended medical visual question answering. GRASP-trained InternVL3-8B and Qwen3-VL-8B outperformed 10 compared general and medical vision-language models on an out-of-distribution benchmark spanning eight imaging modalities. Tuning only 1% of parameters with LoRA was sufficient for GRASP to reach that performance in the study. This suggests that careful post-training aimed at generalisation, rather than larger models or more labelled reasoning traces, may be where clinical gains are currently available. m:…3617

Taken together, these preprints argue for a different procurement and deployment checklist. Ask how a model performs on external datasets from sites like yours, how it behaves when an acquisition artefact is occluded, what its false positive rate is at the sensitivity you actually plan to use, and whether its efficiency fits the hardware where it will run. Accuracy on a public benchmark answers none of those questions on its own. The teams that will avoid expensive clinical disappointments are the ones that test for domain shift, artefacts and operating point before deployment, not after. m:…7001 m:…9988 m:…3617

Limits of this evidence

All four papers are preprints, and their benchmarks differ in task, modality and evaluation design, so they do not support a single ranking of models across medicine. The gastrointestinal, retinal and chest results each rest on specific datasets and artefacts, and the GRASP result rests on one out-of-distribution benchmark design. Local validation on a deployment site's own cases remains the step that cannot be skipped, however clean the published benchmark looks. m:…7001 m:…9988 m:…3617