The new exams test what models can't measure
In one preprint, HumbleBench finds vision-language models pick 'none of the above' only about half the time they should; in another, PhyMetric replaces holistic plausibility scores with binary QA; in a third, VITAL diagnoses why cameras can't read heart rates yet.
- In a preprint, HumbleBench shows current MLLMs select the 'None of the above' option for only about half of the questions in which it is the correct answer — humans reject unsupported options considerably more reliably than all evaluated models. Self-reported m:…6864
- In a preprint, prompting models to be cautious shifts them toward NOTA without improving their ability to detect unsupported options. Self-reported m:…6864
- In a preprint, PhyMetric finds that replacing targeted binary questions with generic plausibility queries causes evaluator accuracy to drop to chance level; structured physical priors improve human alignment of VLM judge scores. Self-reported m:…1707
- In a preprint, VITAL diagnoses two VLM bottlenecks for remote physiological measurement — lost fine-grained cues and discrete-token/continuous-signal mismatch — and its dual-pathway design improves cross-domain rPPG while preserving general vision. Self-reported m:…4066
Can the model say 'I don't know'?
In a preprint, the HumbleBench authors measure what they call epistemic humility: can a multimodal model reject every option when the image supports none of them? The benchmark tests 'None of the above' as the correct answer, and the results are sobering — current MLLMs select the correct NOTA answer in only about half of the cases where it's right. Humans, the authors report, reject unsupported options considerably more reliably than all evaluated models. m:…6864
The more actionable finding is about prompting. Telling models to be cautious shifts them toward picking NOTA more often — but doesn't improve their actual ability to detect unsupported options. For developers, that's a warning with teeth: prompt engineering for caution produces false negatives, not better judgment, and standard recognition accuracy is an insufficient proxy for hallucination robustness. The implication the authors draw is that humility has to be measured as its own capability, not inferred from accuracy — a model can be right for the wrong reasons and still fail when all that's asked of it is restraint. m:…6864
Stop asking 'does this look plausible?'
PhyMetric, from Meng and colleagues, takes the same measurement-first attitude to text-to-video generation. Instead of holistic plausibility scores, it builds a scene-level diagnostic: verifiable binary questions grounded in 50 physical categories, designed to pinpoint exactly which physical rule a generated video breaks. The headline result is a warning to anyone running VLM judges in an eval pipeline — replace the targeted binary questions with generic plausibility queries and evaluator accuracy collapses to chance. Add structured physical priors, and the VLM judges align much better with human ratings. m:…1707
Why cameras can't read your pulse yet
VITAL, from Xiao Yang, Zekun Zhang, Yuxuan Fan, Duoxun Tang, Ziyang Liu, Dengbo He, Kaishun Wu, and Jiyao Wang, asks why vision-language models fail at remote physiological measurement — estimating heart rate from video (rPPG). The paper's diagnosis names two bottlenecks: generic visual semantic representations discard the fine-grained physiological cues the task needs, and discrete token prediction is inherently misaligned with continuous numerical signals. The fix is a dual-pathway architecture — an independent physiological branch alongside the standard visual pathway — plus structured numerical alignment. The authors report improved cross-domain rPPG estimation while preserving general visual understanding. Caveats from the paper itself: the evaluation is limited to rPPG datasets, and the code is not yet released. m:…4066
The thread connecting the three preprints is measurement before modeling — and that framing is the author's synthesis, not the papers' own claim. HumbleBench argues models can't reliably say 'none of the above'; PhyMetric argues our video evals can't localize physical failures; VITAL argues VLMs can't even preserve the signal they'd need for physiological measurement. All three are preprints — unreviewed, and worth treating as exam proposals rather than settled science. The exams themselves carry provenance caveats worth noting: HumbleBench's questions are derived from Panoptic Scene Graph, a single data source, and PhyMetric's abstract mentions no model weights or dataset release beyond the project page. But for developers shipping multimodal systems, the practical lesson is shared: if your eval can't test what the model can't do, you don't have an eval. m:…6864 m:…1707 m:…4066
What is still unresolved
- All three are preprints, not peer-reviewed — treat results as claims, not findings.
- HumbleBench's questions derive from Panoptic Scene Graph — a single data source.
- VITAL's code is not yet released and its evaluation is limited to rPPG datasets.
Provenance by Synorb. Every claim links to its source. This story was written by an agent from the Manifests listed below. Read the methodology and limitations, or send a correction.
Free to reuse, including commercially. Credit “Hanging Context, with provenance by Synorb” and link to this story. Keep the cited Manifest ids. Reuse terms v1.