The shortcuts in medical evidence have measurable gaps

Three shortcuts, three measured gaps

Three preprints posted this week look at three of the faster tools medicine now leans on: AI-assisted systematic reviews, rapid PCR typing of drug resistance, and software that reads hardship out of clinical notes. Each paper measures what its shortcut leaves behind, and the gaps are specific enough to count. m:…7688 m:…5147 m:…6550

All three are preprints, so the numbers are the authors' own and have not been through peer review. Read them as a measurement of the tools, not as a verdict on any single review, test, or patient record. m:…7688 m:…5147 m:…6550

AI-assisted reviews score lower, except where they don't

The first preprint is a matched meta-research study of 231 reviews, comparing AI-assisted systematic reviews with traditional ones. In a preprint, the authors report that AI-assisted reviews scored 1.64 points lower on the 13-item AMSTAR-2 methodological-adherence score, and that PRISMA reporting adherence was 4.7 percentage points lower among AI-assisted reviews in primary adjusted analyses. m:…7688

Only 9.1 percent of AI-assisted reviews reported data-governance or ethics considerations. That is a reporting gap with a practical edge: a review assembled with AI tools that never says how the data was governed leaves its reader unable to check the work. m:…7688

The picture is not one-sided. In the same study, 75.0 percent of AI-assisted reviews were rated low risk on ROBIS, compared with 65.6 percent of non-AI comparators. So the AI-assisted reviews look worse on method and reporting, and better on risk of bias. The split suggests the tools speed up assembly while the quality checks around them have not kept pace. m:…7688

A fast test that misses some resistant cases

The second preprint tests a shortcut of a different kind: real-time PCR for quinolone resistance mutations in Neisseria gonorrhoeae, the bacterium that causes gonorrhoea, as a faster alternative to whole-genome sequencing. The study analysed 235 specimens from symptomatic people in the Eastern Cape, South Africa. In a preprint, the authors report that PCR for gyrA and parC mutations showed high concordance with whole-genome sequencing, 97.0 percent and 81.2 percent respectively. m:…5147

But targeted markers failed to identify resistance in 4 of 65 ciprofloxacin-resistant isolates. The test agrees with sequencing most of the time, and still misses resistance in a small share of the cases that matter most, the ones where the drug will not work. m:…5147

The same paper reports that NG-MAST sequence type ST21253 is the predominant strain in the Eastern Cape, and that macrolide resistance is rare in the studied cohort. The authors' own summary is that targeted PCR is a scalable alternative to sequencing for monitoring quinolone resistance, while phenotypic testing remains necessary for clinical accuracy. m:…5147

The gap is the point. A surveillance tool that misses 4 of 65 resistant cases is useful for tracking a population and risky as a sole basis for treating an individual, and the paper draws exactly that line. m:…5147

What the notes leave out

The third preprint applies natural-language software to 39.7 million clinical notes, looking for stressful life events. In a preprint, the authors report that stressful life events can be identified from clinical notes with high accuracy, with F1 scores of 0.81 to 0.95. m:…6550

The number of different life events is associated with a higher risk of anxiety, a hazard ratio of 2.08 with a 95 percent confidence interval of 2.03 to 2.14, and of depressive disorders, at 1.99 with an interval of 1.94 to 2.03. The associations are strong, and they run in the direction you would expect: more kinds of hardship, more risk. m:…6550

But the record itself is incomplete. Violence and financial distress are documented substantially less often in clinical settings than national estimates suggest. The tool can only read what clinicians wrote down, so the under-recording is a property of the record, not of the software. m:…6550

The result is a measure of hardship that is both powerful and partial: strong enough to predict risk, and blind to the events least likely to make it into the notes. m:…6550

The pattern

Across the three papers, the shortcut is never the problem and never the solution. AI assistance makes reviews faster and thinner; targeted PCR makes resistance typing faster and occasionally blind; note-mining makes hardship visible at a scale no human reader could match, and reproduces the blind spots of the notes themselves. In each case the gap is measurable, 1.64 points, 4 of 65 cases, a category of hardship missing from the record, which is what makes it manageable. m:…7688 m:…5147 m:…6550

The practical move is to keep the shortcut and mind the gap: check AI-assisted reviews against reporting standards, keep phenotypic testing alongside PCR where treatment turns on the result, and treat note-derived hardship as a floor, not a census. Each paper gives its readers the size of the gap, which is the first thing you need before you can close it. m:…7688 m:…5147 m:…6550