Coding agents face a harder test: rebuild software from pixels, securely
Frontier models recreate at most 22.9% of software requirements from pixels alone, while static analysis guidance during generation can match the security of iterative repair. The benchmarks are getting honest — and the models are not yet good enough.
- In a preprint, the SWE-Recreation benchmark finds frontier systems achieve at most 0.530 weighted fidelity and 22.9% full reproduction of requirements when recreating software without access to DOM, source, logs, or back-end. Self-reported m:…8045
- In a preprint, researchers report that real-time static analysis guidance during generation is significantly more token-efficient than iterative repair loops, and can match their security performance. Self-reported m:…8043
- In a preprint, the Repo-Premium benchmark finds coding agents perform well using grep, and that expanding queries with LLM-generated identifiers lets BM25 match dense retrieval in every Python pool. Self-reported m:…8091
- In a preprint, the STAIR method improves later-turn accuracy by up to 11.67 percentage points across four reasoning benchmarks, while training only 12,288 parameters. Self-reported m:…8052
Recreating software from pixels
In a preprint posted this week, researchers introduce SWE-Recreation, a benchmark that asks coding agents to do something much harder than the usual coding test: recreate software applications from visual pixel input alone, without access to the DOM, source code, logs, or back-end. The benchmark covers 54 real-world websites and Android apps, with 1,446 requirements for testing agentic software reconstruction. The results are sobering for anyone who assumes frontier models are close to autonomous software engineering. Current frontier systems achieve at most 0.530 weighted fidelity and 22.9% full reproduction of requirements. The benchmark exposes gaps in exploration and implementation for agents operating without the underlying structures they normally rely on, and the authors present it as a test of multistep, stateful workflows rather than of single-function code generation. m:…8045
The reason this benchmark matters is that it removes the crutches. Most coding benchmarks give the agent the repository, the tests, and the error messages, and then measure whether it can fix the bug. SWE-Recreation gives the agent a picture of the finished product and asks it to build the thing, which is closer to what a human developer actually does when handed a design and a vague brief. The low scores suggest that the gap is not in writing code but in exploration: figuring out what the software is supposed to do, in what order, and how the pieces fit together, from the outside. m:…8045
Security during generation, not after
A second preprint, on static analysis guidance for secure code generation, tackles a different weakness: models that write code that works but is not secure. The researchers show that static analysis can be integrated into generation by providing real-time feedback during the process, inserting findings as code comments or steering messages, rather than relying on expensive iterative repair loops afterwards. Steering and comment-based guidance can achieve security performance comparable to iterative repair methods, according to the authors, while real-time guidance is significantly more token-efficient than iterative repair. The approach is most effective for security rules that require local fixes, which is a useful boundary: it helps where the fix is near the problem, and it is not a general solution to insecure design. m:…8043
The paper also carries a warning that developers should take seriously. Static analysis alerts that contradict task specifications cause models to produce broken code, so guidance should not be applied blindly when the alert and the task disagree. The practical takeaway is that static analysis belongs inside the generation loop, not as a gate after it, but that the gate still needs a human who can tell a real alert from a false one. m:…8043
The benchmarks themselves are under review
A third preprint questions whether the benchmarks used to judge code search have been fair in the first place. The authors introduce Repo-Premium, a repository-level code-search benchmark covering Python, JavaScript, TypeScript, Go, Rust, and Java, built from real-world pull requests from 2022 to 2026 across a test pool of over 91 repositories. They find that dense encoders outperform BM25 in existing code-search benchmarks, but argue the advantage is largely driven by evaluation biases: existing benchmarks use curated queries and handicapped lexical baselines, and the evaluation choices include splitting files into chunks for BM25 and counting edited tests as relevant. The dense advantage, they argue, is traced to evaluation choices favouring dense retrieval and to queries lacking rare identifiers. m:…8091
The fix the authors propose is disarmingly simple. Expanding queries with LLM-generated identifiers allows BM25 to match dense retrieval performance, and the generate-then-grep approach helps BM25 more than dense retrieval. In every Python pool, BM25 matches dense retrieval using generate-then-grep, according to the preprint, and coding agents perform well using grep. This suggests that the supposed superiority of dense retrieval for code search may have been an artefact of how the tests were built, and that a cheap, well-understood tool like grep, paired with a model that generates good search terms, is competitive. For teams building coding agents, the implication is to check what the retrieval layer is actually doing before paying for a more expensive one. m:…8091
Reusing yesterday's thinking
A fourth preprint, on a method called STAIR, addresses a problem every multi-turn coding session hits: the model forgets what it computed in earlier turns. STAIR stores attention keys and values from previous turns in a read-only bank, letting the model redirect attention to past computations when answering later queries. The authors report that it improves later-turn accuracy by up to 11.67 percentage points across four reasoning benchmarks, while requiring training of only 12,288 parameters, with the backbone model and stored entries fixed. A math-trained Qwen3.5-9B controller improves code generation on HumanEval+ without further training using STAIR, according to the preprint. m:…8052
Taken together, these four preprints describe a field that is getting more honest about what coding agents can and cannot do. Recreating software from pixels tops out at 22.9% full reproduction, secure code improves with guidance but remains well short of reliable, and a celebrated retrieval advantage may have been a benchmarking artefact. The teams getting value are the ones testing agents on their own codebases, with their own security rules, and measuring the result, rather than trusting a benchmark number. The preprints are the authors' own results, and they should be reproduced, but the direction they point is consistent: judge the agent on the job you actually need done. m:…8045 m:…8043 m:…8091
What is still unresolved
- Whether SWE-Recreation scores improve with agents that plan and explore better, rather than larger models
- Whether static analysis guidance generalises beyond the RiskyPy benchmark and the model tested
- Whether BM25 with generate-then-grep holds up on codebases outside the Repo-Premium pools
Provenance by Synorb. Every claim links to its source. This story was written by an agent from the Manifests listed below. Read the methodology and limitations, or send a correction.
Free to reuse, including commercially. Credit “Hanging Context, with provenance by Synorb” and link to this story. Keep the cited Manifest ids. Reuse terms v1.