Stop Asking Vision Models to Guess. Make Them Ground and Execute.

Execution beats generation

In a preprint called VisualCoding, researchers shifted visual reasoning from direct generation to execution-guided program acquisition, and found that approach improved visual reasoning reliability compared with standard prompting. Reusing frozen visual solvers from larger models significantly boosted performance in smaller models, and direct execution of visual programs was superior to textual or callable reuse for maintaining answer validity. The distinction matters: a program that runs and can be checked is different from an answer that merely reads well, and smaller models benefited most when they could reuse executed work from larger models rather than re-deriving an answer from scratch. m:…0279

A robotics benchmark preprint shows what happens without that grounding. In BEHAVIOR-Spatial, current vision-language-action models fell from 41.6% to 16.1% when instructions changed within a fixed layout. Oracle visual target cues improved performance on held-out configurations to 60.3% and on changed instructions to 38.0%, and there was a policy-dependent correlation between vision-language question-answering and grounding accuracy and vision-language-action manipulation performance. In plain terms, the robots were not failing because the rooms changed. They were failing because the instruction changed, which suggests they had memorised tasks rather than understood spatial language. m:…3514

Use the evidence requested

A third preprint, on modality control, names a related failure. Accuracy alone was insufficient to measure vision-language grounding because models may rely on parametric knowledge rather than requested evidence. The authors introduced Flip and Stay metrics to measure control through counterfactual interventions, found that Counterfactual Twin-Margin supervision improved control in settings where standard supervision was weaker, and found that a single learned embedding per modality provided strong lightweight visual control with little degradation on held-out values. For builders, the practical test is counterfactual: change the image or the text and see whether the answer follows the requested evidence, instead of trusting an accuracy score that a model can reach from memory. m:…7885

A fourth preprint shows that fixing the inference process, rather than retraining the model, can be enough. Autonomous evolution of inference programs improved vision-language accuracy on scientific charts without retraining, raising average accuracy from 16.6% to 41.3% and running 3.8 times faster than baseline programs due to simplified answer generation. That combination, higher accuracy and lower latency from evolved programs, is the kind of result that changes deployment economics: teams can improve a specialised task by evolving how the model is used, without paying for a new training run. m:…4696

Taken together, these preprints point to a single engineering discipline. Require visual models to execute checkable programs where possible, test them under changed instructions in fixed layouts, test them counterfactually against the evidence they were asked to use, and evolve the inference pipeline before retraining. A model that passes a static benchmark while failing those tests is not ready for an agent that acts in the world, however fluent its answers look. m:…0279 m:…3514 m:…7885 m:…4696

Limits

All four are preprints, and their benchmarks cover different tasks, from chart analysis to robotic manipulation, so they do not support a single ranking of vision models. The oracle-cue result in particular shows how far current models are from grounding on their own, rather than offering a deployable fix. The next informative evidence would be the same tests run on production agents under changed instructions and counterfactual evidence, not another static leaderboard score. m:…3514 m:…7885