Clinical AI Is Being Graded on Guidelines, and the Grades Do Not Travel Well

Three tests, one pattern

In a preprint, researchers built EM-BENCH, a physician-validated benchmark for emergency-medicine decision making, and found that closed-access models outperformed open-weight models on guideline concordance, that accuracy rose 14 percentage points per tenfold increase in parameters, and that accuracy improved 8.9 percentage points per release year. Inference-time reasoning modes did not consistently improve guideline concordance in the same tests, which cuts against the idea that asking a model to think longer is a reliable fix for clinical accuracy. m:…0920

In a preprint on electronic health record phenotyping, the same pattern showed up in a different form: detailed codebooks took LLM-generated SQL to full executability on UNC Health data, and high-context outputs reached recall above 0.90, beating a diagnosis-code baseline. But limited positive predictive value in real-world data means the outputs still need local validation before they can be trusted in a hospital system. m:…6793

In a preprint on peer review, language models detected major methodological errors in clinical trial reports at rates between 53 and 99 percent, far above the 27 percent rate for untrained human reviewers. The catch is on the other side of the ledger: the models flagged 2 to 29 percent of error types in control reports that contained no inserted errors, so a model that catches more will also send more clean work back for needless rework unless a human stays in the loop. m:…4154

Context is the intervention

Read together, the three preprints point to a single practical conclusion: the context supplied to the model matters at least as much as the model itself. Guideline concordance scaled with size and recency in the EM-BENCH tests, yet the phenotyping work shows that richer database context, in the form of detailed codebooks and sample rows, is what turned unreliable SQL into executable, high-recall queries. For a team deploying clinical AI, that suggests the highest-return work may sit in the data layer, in codebooks, schemas, and local validation sets, rather than in chasing the next model release. m:…0920 m:…6793

The peer-review preprint adds a second lesson about where to place humans. A detection rate up to 99 percent sounds like a replacement for reviewers, but a false-positive range up to 29 percent makes it a triage tool instead: useful for catching what tired or untrained eyes miss, costly if every flag is treated as a finding. The sensible deployment, on the evidence in these preprints, is model-first screening with human adjudication, and with the false-positive rate measured on local reports rather than assumed from a benchmark. m:…4154

The question buyers should ask

None of these preprints reports patient outcomes, and that gap is the second-order question for anyone buying or building clinical AI. Guideline concordance, SQL executability, and error detection are all proxy measures, and each one degraded or wobbled when the setting changed, whether from synthetic to real records or from benchmark vignettes to control reports. A vendor score earned on one dataset is therefore weak evidence about performance on a specific hospital's data, and the preprints themselves point buyers toward the stronger test: run the model on local cases, measure precision as well as recall, and publish the false-positive rate alongside the headline accuracy. m:…6793 m:…4154

For builders, the implication is narrower and more actionable. Benchmarks like EM-BENCH give the field a shared ruler, but the phenotyping and peer-review results argue that the ruler needs a local counterpart: a validation set drawn from the deployment site, with the same codebooks and the same report formats the model will meet in production. Teams that build that local harness first will know sooner whether a new model actually helps, and teams that skip it will keep rediscovering the synthetic-to-real gap after deployment. m:…0920 m:…6793