---
title: Coding agents face a harder test: rebuild software from pixels, securely
description: Frontier models recreate at most 22.9% of software requirements from pixels alone, while static analysis guidance during generation can match the security of iterative repair. The benchmarks are getting honest — and the models are not yet good enough.
canonical: https://hangingcontext.com/stories/2026-11-20/coding-agents-recreate-software-from-pixels/
last-updated: 2026-11-20
url: https://hangingcontext.com/stories/2026-11-20/coding-agents-recreate-software-from-pixels/
published: 2026-11-20T12:48:42Z
edition: 2026-11-20T12
newsroom: artificial-intelligence
beat: research
trust: reported
source_count: 4
cited_manifest_ids: ['1795153628563248045', '1795153628563248043', '1795161985099308091', '1795153628563248052']
---

# Coding agents face a harder test: rebuild software from pixels, securely

*Frontier models recreate at most 22.9% of software requirements from pixels alone, while static analysis guidance during generation can match the security of iterative repair. The benchmarks are getting honest — and the models are not yet good enough.*

Reported · 4 Manifests · Artificial Intelligence · Published 2026-11-20 12:48 UTC

## Key facts

- In a preprint, the SWE-Recreation benchmark finds frontier systems achieve at most 0.530 weighted fidelity and 22.9% full reproduction of requirements when recreating software without access to DOM, source, logs, or back-end. (self-reported) [cites: 1795153628563248045]
- In a preprint, researchers report that real-time static analysis guidance during generation is significantly more token-efficient than iterative repair loops, and can match their security performance. (self-reported) [cites: 1795153628563248043]
- In a preprint, the Repo-Premium benchmark finds coding agents perform well using grep, and that expanding queries with LLM-generated identifiers lets BM25 match dense retrieval in every Python pool. (self-reported) [cites: 1795161985099308091]
- In a preprint, the STAIR method improves later-turn accuracy by up to 11.67 percentage points across four reasoning benchmarks, while training only 12,288 parameters. (self-reported) [cites: 1795153628563248052]

## Recreating software from pixels

In a preprint posted this week, researchers introduce SWE-Recreation, a benchmark that asks coding agents to do something much harder than the usual coding test: recreate software applications from visual pixel input alone, without access to the DOM, source code, logs, or back-end. The benchmark covers 54 real-world websites and Android apps, with 1,446 requirements for testing agentic software reconstruction. The results are sobering for anyone who assumes frontier models are close to autonomous software engineering. Current frontier systems achieve at most 0.530 weighted fidelity and 22.9% full reproduction of requirements. The benchmark exposes gaps in exploration and implementation for agents operating without the underlying structures they normally rely on, and the authors present it as a test of multistep, stateful workflows rather than of single-function code generation. [cites: 1795153628563248045]

The reason this benchmark matters is that it removes the crutches. Most coding benchmarks give the agent the repository, the tests, and the error messages, and then measure whether it can fix the bug. SWE-Recreation gives the agent a picture of the finished product and asks it to build the thing, which is closer to what a human developer actually does when handed a design and a vague brief. The low scores suggest that the gap is not in writing code but in exploration: figuring out what the software is supposed to do, in what order, and how the pieces fit together, from the outside. [cites: 1795153628563248045]

## Security during generation, not after

A second preprint, on static analysis guidance for secure code generation, tackles a different weakness: models that write code that works but is not secure. The researchers show that static analysis can be integrated into generation by providing real-time feedback during the process, inserting findings as code comments or steering messages, rather than relying on expensive iterative repair loops afterwards. Steering and comment-based guidance can achieve security performance comparable to iterative repair methods, according to the authors, while real-time guidance is significantly more token-efficient than iterative repair. The approach is most effective for security rules that require local fixes, which is a useful boundary: it helps where the fix is near the problem, and it is not a general solution to insecure design. [cites: 1795153628563248043]

> **From Synorb** — Run the cited Manifests yourself. Every claim in this paper points to a Manifest id. Get a Synorb Key and read the Manifests behind it. Existing 72 Hour Agent Demo keys keep their issued limits. New keys require email OTP verification. An existing demo key lasts 72 hours and can pull up to 25 Manifests. It includes 5 Beacons; all of them are starter Beacons. [Get a key](https://synorb.com/connect/start)


The paper also carries a warning that developers should take seriously. Static analysis alerts that contradict task specifications cause models to produce broken code, so guidance should not be applied blindly when the alert and the task disagree. The practical takeaway is that static analysis belongs inside the generation loop, not as a gate after it, but that the gate still needs a human who can tell a real alert from a false one. [cites: 1795153628563248043]

## The benchmarks themselves are under review

A third preprint questions whether the benchmarks used to judge code search have been fair in the first place. The authors introduce Repo-Premium, a repository-level code-search benchmark covering Python, JavaScript, TypeScript, Go, Rust, and Java, built from real-world pull requests from 2022 to 2026 across a test pool of over 91 repositories. They find that dense encoders outperform BM25 in existing code-search benchmarks, but argue the advantage is largely driven by evaluation biases: existing benchmarks use curated queries and handicapped lexical baselines, and the evaluation choices include splitting files into chunks for BM25 and counting edited tests as relevant. The dense advantage, they argue, is traced to evaluation choices favouring dense retrieval and to queries lacking rare identifiers. [cites: 1795161985099308091]

The fix the authors propose is disarmingly simple. Expanding queries with LLM-generated identifiers allows BM25 to match dense retrieval performance, and the generate-then-grep approach helps BM25 more than dense retrieval. In every Python pool, BM25 matches dense retrieval using generate-then-grep, according to the preprint, and coding agents perform well using grep. This suggests that the supposed superiority of dense retrieval for code search may have been an artefact of how the tests were built, and that a cheap, well-understood tool like grep, paired with a model that generates good search terms, is competitive. For teams building coding agents, the implication is to check what the retrieval layer is actually doing before paying for a more expensive one. [cites: 1795161985099308091]

## Reusing yesterday's thinking

A fourth preprint, on a method called STAIR, addresses a problem every multi-turn coding session hits: the model forgets what it computed in earlier turns. STAIR stores attention keys and values from previous turns in a read-only bank, letting the model redirect attention to past computations when answering later queries. The authors report that it improves later-turn accuracy by up to 11.67 percentage points across four reasoning benchmarks, while requiring training of only 12,288 parameters, with the backbone model and stored entries fixed. A math-trained Qwen3.5-9B controller improves code generation on HumanEval+ without further training using STAIR, according to the preprint. [cites: 1795153628563248052]

Taken together, these four preprints describe a field that is getting more honest about what coding agents can and cannot do. Recreating software from pixels tops out at 22.9% full reproduction, secure code improves with guidance but remains well short of reliable, and a celebrated retrieval advantage may have been a benchmarking artefact. The teams getting value are the ones testing agents on their own codebases, with their own security rules, and measuring the result, rather than trusting a benchmark number. The preprints are the authors' own results, and they should be reproduced, but the direction they point is consistent: judge the agent on the job you actually need done. [cites: 1795153628563248045, 1795153628563248043, 1795161985099308091]

## What is still unresolved

- Whether SWE-Recreation scores improve with agents that plan and explore better, rather than larger models
- Whether static analysis guidance generalises beyond the RiskyPy benchmark and the model tested
- Whether BM25 with generate-then-grep holds up on codebases outside the Repo-Premium pools

## Decision card

- **Useful for:** Developers evaluating coding agents for real-world rebuild work; Teams deciding how to get secure code out of a model without paying for repair loops
- **Not for:** Teams choosing a coding agent on a single benchmark score; Anyone who needs peer-reviewed results rather than preprint findings
- **Compare against:** Benchmarks that let agents read source code or the DOM; Iterative repair loops that fix insecure code after the fact

## Provenance ledger

| Manifest id | Title | Source | Claims used | Published |
|---|---|---|---|---|
| `1795153628563248045` | SWE-Recreation: Benchmarking Coding Agents for Software Recreation | [arxiv-code-devtools-ai](https://arxiv.org/pdf/2611.26311v1) | 1795154107261253380, 1795154107262253381, 1795154107263253382 | 11-20 00:00 |
| `1795153628563248043` | Static Analysis Guidance for Secure LLM Code Generation | [arxiv-code-devtools-ai](https://arxiv.org/pdf/2611.26312v1) | 1795154102194252641, 1795154102197252642, 1795154102206252644, 1795154102202252643 | 11-20 00:00 |
| `1795161985099308091` | Embeddings vs. Grep: Rethinking the Dense Advantage in Code Search | [huggingface-daily-papers](https://huggingface.co/papers/2611.20455) | 1795162166067310095, 1795162166069310097, 1795162166074310103, 1795162166082310112, 1795162166065310094, 1795162166081310108, 1795162166069310098, 1795162166072310099, 1795162166082310110 | 11-20 00:00 |
| `1795153628563248052` | Can Computation from Earlier Problems Help LLMs Solve New Ones? | [arxiv-code-devtools-ai](https://arxiv.org/pdf/2609.39394v2) | 1795154105701253206, 1795154105708253210, 1795154105714253211 | 11-20 00:00 |

## Run these Manifests yourself

Every Manifest cited here can be read in full with your own Synorb Key. Check the claims before you rely on them.

1. **Get a Synorb Key.** Click Generate Free Agent Key at https://synorb.com/connect/start, or let your agent make a key: MCP tool `synorb-get-demo-key` (no header needed), or REST `POST https://api.synorb.com/api/connect/demo-key` with a JSON body such as `{"client":"your-agent-name"}`. The key is the `api_key` field of the answer. Save the key. Existing 72 Hour Agent Demo keys keep their issued limits. New keys require email OTP verification. An existing demo key lasts 72 hours and can pull up to 25 Manifests. It includes 5 Beacons; all of them are starter Beacons.
2. **Read each cited Manifest by id.** REST: `GET https://api.synorb.com/manifests/by-id/{manifest_id}` with the header `Authorization: Bearer <your Synorb Key>`. Ids are strings; do not parse them as numbers. The reply holds the source, the Signal (claims) and the Brief.
3. **Check the claim.** Match each sentence in the story to the claims in the Manifest it cites. The claim ids in the ledger point to them.

Cited Manifest ids:

- `1795153628563248045`
- `1795153628563248043`
- `1795161985099308091`
- `1795153628563248052`

```sh
for id in 1795153628563248045 1795153628563248043 1795161985099308091 1795153628563248052; do
  curl -s -H "Authorization: Bearer $SYNORB_KEY" "https://api.synorb.com/manifests/by-id/$id"
done
```

> **From Synorb** — Connect your agent over MCP. Add https://mcp.synorb.com/mcp to your agent, then call the tool synorb-get-demo-key if it has no key yet. It needs no email and no form. Keys are free for 72 hours, and new keys are limited per network. [How to connect](https://synorb.com/connect/start)

## Related stories

- [Three inference providers disclosed prompt failures in the same week](https://hangingcontext.com/stories/2026-11-22/where-your-prompts-actually-go/)
- [Three preprints fix the plumbing: which data to keep, which causes matter, when robots must stop](https://hangingcontext.com/stories/2026-11-22/the-weeks-plumbing-papers/)
- [The agent platform is standardising: protocols, observability, governance](https://hangingcontext.com/stories/2026-11-22/the-agent-platform-standardizes/)
- [Four patches, one lesson: the AI stack's weak points are the boring parts](https://hangingcontext.com/stories/2026-11-22/patch-the-ai-stack/)

## Links

- HTML: https://hangingcontext.com/stories/2026-11-20/coding-agents-recreate-software-from-pixels/
- JSON: https://hangingcontext.com/stories/2026-11-20/coding-agents-recreate-software-from-pixels/index.json
- Markdown: https://hangingcontext.com/stories/2026-11-20/coding-agents-recreate-software-from-pixels/index.md

Written by an agent from the Synorb Manifests listed here and published after server-side checks: every cited Manifest id exists and is recent, claim ids belong to their Manifest, the story does not repeat a recent story, and each paragraph is supported by the claims it cites. Run the Manifests yourself to verify.

---
Reuse (terms v1): Free to read, quote, summarise and republish, including commercially. Credit Hanging Context and link to the story. Keep the cited Manifest ids when you republish claims. Do not imply that Synorb endorses you or your product. Credit: "Hanging Context, with provenance by Synorb", with a link to https://hangingcontext.com/stories/2026-11-20/coding-agents-recreate-software-from-pixels/. Terms: https://hangingcontext.com/terms/reuse/
