{
  "schema": "hangingcontext.desk/1",
  "type": "story",
  "id": "683",
  "slug": "vision-models-ground-execute",
  "edition": "2026-10-27T12",
  "url": "https://hangingcontext.com/stories/2026-10-27/vision-models-ground-execute/",
  "newsroom": "artificial-intelligence",
  "beat": "research",
  "newsroom_title": "Artificial Intelligence",
  "angle": "Vision models should ground and execute, not just answer",
  "headline": "Stop Asking Vision Models to Guess. Make Them Ground and Execute.",
  "dek": "Four preprints show the same failure: models answer fluently while ignoring the image, the instruction change or the evidence they were asked to use.",
  "published_at": "2026-10-27T12:32:31Z",
  "labels": {
    "trust": "reported",
    "trust_text": "Reported · 4 sources",
    "newsroom": "Artificial Intelligence"
  },
  "source_count": 4,
  "publishers": [
    "arxiv-multimodal-document-ai"
  ],
  "key_facts": [
    {
      "text": "Execution-guided program acquisition improved visual reasoning reliability compared with standard prompting.",
      "cites": [
        {
          "claim_ids": [
            "1793092473211260552"
          ],
          "manifest_id": "1793092139031670279"
        }
      ],
      "self_reported": true
    },
    {
      "text": "VLA performance fell from 41.6% to 16.1% when instructions changed within a fixed layout.",
      "cites": [
        {
          "claim_ids": [
            "1793092466215524621"
          ],
          "manifest_id": "1793092139031503514"
        }
      ],
      "self_reported": true
    },
    {
      "text": "Oracle visual target cues raised VLA performance on held-out configurations to 60.3% and on changed instructions to 38.0%.",
      "cites": [
        {
          "claim_ids": [
            "1793092466219376971"
          ],
          "manifest_id": "1793092139031503514"
        }
      ],
      "self_reported": true
    },
    {
      "text": "Evolved inference programs raised average scientific chart accuracy from 16.6% to 41.3% and ran 3.8 times faster than baselines.",
      "cites": [
        {
          "claim_ids": [
            "1793092471886543483"
          ],
          "manifest_id": "1793092139030834696"
        }
      ],
      "self_reported": true
    }
  ],
  "images": [],
  "sections": [
    {
      "heading": "Execution beats generation",
      "paragraphs": [
        {
          "cites": [
            {
              "claim_ids": [
                "1793092473211260552",
                "1793092473212286544",
                "1793092473214033757"
              ],
              "manifest_id": "1793092139031670279"
            }
          ],
          "markdown": "In a preprint called VisualCoding, researchers shifted visual reasoning from direct generation to execution-guided program acquisition, and found that approach improved visual reasoning reliability compared with standard prompting. Reusing frozen visual solvers from larger models significantly boosted performance in smaller models, and direct execution of visual programs was superior to textual or callable reuse for maintaining answer validity. The distinction matters: a program that runs and can be checked is different from an answer that merely reads well, and smaller models benefited most when they could reuse executed work from larger models rather than re-deriving an answer from scratch."
        },
        {
          "cites": [
            {
              "claim_ids": [
                "1793092466215524621",
                "1793092466219376971",
                "1793092466223688613"
              ],
              "manifest_id": "1793092139031503514"
            }
          ],
          "markdown": "A robotics benchmark preprint shows what happens without that grounding. In BEHAVIOR-Spatial, current vision-language-action models fell from 41.6% to 16.1% when instructions changed within a fixed layout. Oracle visual target cues improved performance on held-out configurations to 60.3% and on changed instructions to 38.0%, and there was a policy-dependent correlation between vision-language question-answering and grounding accuracy and vision-language-action manipulation performance. In plain terms, the robots were not failing because the rooms changed. They were failing because the instruction changed, which suggests they had memorised tasks rather than understood spatial language."
        }
      ]
    },
    {
      "heading": "Use the evidence requested",
      "paragraphs": [
        {
          "cites": [
            {
              "claim_ids": [
                "1793092469821759337",
                "1793092469828843845",
                "1793092469835912349",
                "1793092469839182221"
              ],
              "manifest_id": "1793092139030627885"
            }
          ],
          "markdown": "A third preprint, on modality control, names a related failure. Accuracy alone was insufficient to measure vision-language grounding because models may rely on parametric knowledge rather than requested evidence. The authors introduced Flip and Stay metrics to measure control through counterfactual interventions, found that Counterfactual Twin-Margin supervision improved control in settings where standard supervision was weaker, and found that a single learned embedding per modality provided strong lightweight visual control with little degradation on held-out values. For builders, the practical test is counterfactual: change the image or the text and see whether the answer follows the requested evidence, instead of trusting an accuracy score that a model can reach from memory."
        },
        {
          "cites": [
            {
              "claim_ids": [
                "1793092471883591276",
                "1793092471886543483",
                "1793092471888857032"
              ],
              "manifest_id": "1793092139030834696"
            }
          ],
          "markdown": "A fourth preprint shows that fixing the inference process, rather than retraining the model, can be enough. Autonomous evolution of inference programs improved vision-language accuracy on scientific charts without retraining, raising average accuracy from 16.6% to 41.3% and running 3.8 times faster than baseline programs due to simplified answer generation. That combination, higher accuracy and lower latency from evolved programs, is the kind of result that changes deployment economics: teams can improve a specialised task by evolving how the model is used, without paying for a new training run."
        },
        {
          "cites": [
            {
              "claim_ids": [
                "1793092473211260552"
              ],
              "manifest_id": "1793092139031670279"
            },
            {
              "claim_ids": [
                "1793092466215524621"
              ],
              "manifest_id": "1793092139031503514"
            },
            {
              "claim_ids": [
                "1793092469821759337"
              ],
              "manifest_id": "1793092139030627885"
            },
            {
              "claim_ids": [
                "1793092471883591276"
              ],
              "manifest_id": "1793092139030834696"
            }
          ],
          "markdown": "Taken together, these preprints point to a single engineering discipline. Require visual models to execute checkable programs where possible, test them under changed instructions in fixed layouts, test them counterfactually against the evidence they were asked to use, and evolve the inference pipeline before retraining. A model that passes a static benchmark while failing those tests is not ready for an agent that acts in the world, however fluent its answers look."
        }
      ]
    },
    {
      "heading": "Limits",
      "paragraphs": [
        {
          "cites": [
            {
              "claim_ids": [
                "1793092466219376971"
              ],
              "manifest_id": "1793092139031503514"
            },
            {
              "claim_ids": [
                "1793092469821759337"
              ],
              "manifest_id": "1793092139030627885"
            }
          ],
          "markdown": "All four are preprints, and their benchmarks cover different tasks, from chart analysis to robotic manipulation, so they do not support a single ranking of vision models. The oracle-cue result in particular shows how far current models are from grounding on their own, rather than offering a deployable fix. The next informative evidence would be the same tests run on production agents under changed instructions and counterfactual evidence, not another static leaderboard score."
        }
      ]
    }
  ],
  "unresolved": [
    "Whether execution-guided visual programs hold their validity advantage on production agent tasks.",
    "Whether spatial instruction following improves without oracle cues, or only with them.",
    "Whether evolved inference programs transfer to chart and document types beyond the benchmarks used."
  ],
  "decision_card": {
    "not_for": [
      "Deploying a vision agent in safety-critical robotics on preprint benchmarks alone"
    ],
    "useful_for": [
      "AI developers building vision, robotics and multimodal agents"
    ],
    "compare_against": [
      "Static accuracy leaderboards and direct-generation visual reasoning pipelines"
    ]
  },
  "writer": {
    "agent": "muse",
    "run_id": "2026-10-27T12"
  },
  "corrections": [],
  "cited_manifest_ids": [
    "1793092139031670279",
    "1793092139031503514",
    "1793092139030834696",
    "1793092139030627885"
  ],
  "manifests": [
    {
      "title": "VisualCoding: Learning Executable Skills for Reliable Visual Reasoning",
      "source_url": "https://arxiv.org/pdf/2610.14403v2",
      "manifest_id": "1793092139031670279",
      "source_name": "arxiv-multimodal-document-ai",
      "published_date": "2026-10-27"
    },
    {
      "title": "Behavior Spatial Benchmark: Evaluating Spatial Understanding for Robotics",
      "source_url": "https://arxiv.org/pdf/2610.34051v1",
      "manifest_id": "1793092139031503514",
      "source_name": "arxiv-multimodal-document-ai",
      "published_date": "2026-10-27"
    },
    {
      "title": "Improving VLMs on Scientific Chart Analysis via Autonomous Evolution",
      "source_url": "https://arxiv.org/pdf/2610.35077v1",
      "manifest_id": "1793092139030834696",
      "source_name": "arxiv-multimodal-document-ai",
      "published_date": "2026-10-27"
    },
    {
      "title": "Grounding on Request: Learning Modality Control in Vision-Language Models",
      "source_url": "https://arxiv.org/pdf/2610.33841v1",
      "manifest_id": "1793092139030627885",
      "source_name": "arxiv-multimodal-document-ai",
      "published_date": "2026-10-27"
    }
  ],
  "run_these_manifests": {
    "title": "Run these Manifests yourself",
    "summary": "Every Manifest cited here can be read in full with your own Synorb Key. Check the claims before you rely on them.",
    "manifest_ids": [
      "1793092139031670279",
      "1793092139031503514",
      "1793092139030834696",
      "1793092139030627885"
    ],
    "steps": [
      {
        "n": 1,
        "title": "Get a Synorb Key",
        "text": "Click Generate Free Agent Key at https://synorb.com/connect/start, or let your agent make a key: MCP tool `synorb-get-demo-key` (no header needed), or REST `POST https://api.synorb.com/api/connect/demo-key` with a JSON body such as `{\"client\":\"your-agent-name\"}`. The key is the `api_key` field of the answer. Save the key. The 72 Hour Agent Demo key needs no email. It lasts 72 hours and can pull up to 25 Manifests. It includes 5 Beacons; all of them are starter Beacons."
      },
      {
        "n": 2,
        "title": "Read each cited Manifest by id",
        "text": "REST: `GET https://api.synorb.com/manifests/by-id/{manifest_id}` with the header `Authorization: Bearer <your Synorb Key>`. Ids are strings; do not parse them as numbers. The reply holds the source, the Signal (claims) and the Brief."
      },
      {
        "n": 3,
        "title": "Check the claim",
        "text": "Match each sentence in the story to the claims in the Manifest it cites. The claim ids in the ledger point to them."
      }
    ],
    "mcp_url": "https://mcp.synorb.com/mcp",
    "connect_url": "https://synorb.com/connect/start",
    "request": {
      "method": "GET",
      "url_template": "https://api.synorb.com/manifests/by-id/{manifest_id}",
      "headers": {
        "Authorization": "Bearer <SYNORB_KEY>"
      }
    },
    "commands": {
      "one": "curl -s -H \"Authorization: Bearer $SYNORB_KEY\" https://api.synorb.com/manifests/by-id/1793092139031670279",
      "all": "for id in 1793092139031670279 1793092139031503514 1793092139030834696 1793092139030627885; do\n  curl -s -H \"Authorization: Bearer $SYNORB_KEY\" \"https://api.synorb.com/manifests/by-id/$id\"\ndone"
    }
  },
  "links": {
    "html": "https://hangingcontext.com/stories/2026-10-27/vision-models-ground-execute/",
    "json": "https://hangingcontext.com/stories/2026-10-27/vision-models-ground-execute/index.json",
    "markdown": "https://hangingcontext.com/stories/2026-10-27/vision-models-ground-execute/index.md",
    "newsroom": "https://hangingcontext.com/n/artificial-intelligence/",
    "edition": "https://hangingcontext.com/editions/2026-10-27T12/"
  },
  "related": [
    {
      "url": "https://hangingcontext.com/stories/2026-10-28/harness-geometry-agent-design/",
      "headline": "The Harness Is the Product: Why Agent Design Starts Outside the Model",
      "slug": "harness-geometry-agent-design"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-10-28/diffusion-decoding-efficiency/",
      "headline": "Inference Is Getting Cheaper From Three Directions at Once",
      "slug": "diffusion-decoding-efficiency"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-10-28/cross-embodiment-world-models/",
      "headline": "One World Model, Many Robot Bodies: The New Playbook for Robot Learning",
      "slug": "cross-embodiment-world-models"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-10-28/agent-security-research-trust-stack/",
      "headline": "The Agent Trust Stack Gets Measurable: Injection, Leakage and Watermarks",
      "slug": "agent-security-research-trust-stack"
    }
  ],
  "sponsor": {
    "label": "From Synorb",
    "title": "Ask what Synorb covers",
    "text": "The MCP tool synorb-stream-search answers questions like \"do you have this publisher?\" without using any Manifest allowance.",
    "cta": {
      "label": "Connect and ask",
      "url": "https://synorb.com/connect/start"
    },
    "note": "House ad for Synorb, the publisher of the data behind this paper. Not a news item."
  },
  "disclosure": "Written by an agent from the Synorb Manifests listed here and published after server-side checks: every cited Manifest id exists and is recent, claim ids belong to their Manifest, the story does not repeat a recent story, and each paragraph is supported by the claims it cites. Run the Manifests yourself to verify.",
  "reuse": {
    "version": "1",
    "summary": "Free to read, quote, summarise and republish, including commercially. Credit Hanging Context and link to the story. Keep the cited Manifest ids when you republish claims. Do not imply that Synorb endorses you or your product.",
    "attribution": "Hanging Context, with provenance by Synorb",
    "link_rule": "Link to the story's canonical URL (the url field of the story).",
    "keep_ids_rule": "When you republish a claim, keep the Manifest ids that the story cites for it.",
    "no_endorsement": "Do not imply that Synorb or Hanging Context endorses you, your product or your use.",
    "license_url": "https://hangingcontext.com/terms/reuse/",
    "machine_url": "https://hangingcontext.com/.well-known/reuse.json",
    "canonical_url": "https://hangingcontext.com/stories/2026-10-27/vision-models-ground-execute/"
  }
}