When requirements interact, coding agents stumble, and new benchmarks are built to catch it

The failure mode is interaction

Long-horizon coding agents fail less on single tasks than on requirements that interact, and a preprint builds training and evaluation around exactly that gap. CodeSpecWeave synthesizes complex, interdependent software development tasks with verified tests to train coding agents. Current coding agent work often emphasizes primary goals while ignoring complex constraints, the authors argue, and lacks specification in both quantity and diversity. The point is familiar to anyone who has shipped software: a feature that satisfies the headline requirement while breaking a security constraint, a performance budget, or an interface contract is not done. m:…5539

The pipeline has two roles. A synthesis agent iteratively develops coupled task requirements and executable tests, and a review agent audits solution trajectories from multiple models. The pipeline constructed 4,670 training tasks across 2,865 repositories, and reinforcement learning on those tasks improves performance on DeepSWE-v1.1, the authors report. The accompanying benchmark, CodeSpecWeave-Bench, comprises 100 tasks and 1,044 specifications, and reveals significant headroom in handling interdependent engineering constraints. Headroom is the honest word: the benchmark exists because current agents still leave room on the table. m:…5539

Judgment is the other gap

A second preprint measures a parallel gap in research agents: generating options is easier than ranking them. Six frontier models were tested as research agents using seven recent NLP papers, proposing 415 minimum viable experiments in total. Of those, 194 were judged sound, valuable, and new, and only 73 were judged beyond the seed paper and non-obvious. Agent rankings did not show improvement over choosing the experiment written first, and agent forecasts beat a base rate but were not measurably more accurate than a fresh instance of the same model given only the paper. Agent first choices matched the auditors most valued experiment more often than a random pick, but that is a low bar for a system sold on judgment. m:…6991

Multi-agent teams cost more without a clear overall improvement in proposal quality, the study finds. The parallel with coding agents is direct. A system that can produce a plausible solution, or a plausible experiment, is not the same as a system that can choose well among competing constraints and candidates. Evaluation that only counts plausible outputs will overrate both. Evaluation that forces ranking, constraint satisfaction, and trade-offs is where the real gaps show up. m:…6991

Safety and training honesty

A third preprint moves evaluation into training data itself. SimpleLoRIF uses gradient alignment to filter instruction data that silently degrades language model safety, restricting backpropagation to the final four transformer blocks to enable scaling to millions of examples. SafeTulu is a version of Tulu 3 with the 20 percent of examples most gradient-aligned with unsafe responses removed, and training on SafeTulu reduces GCG attack success rate by up to 22 percent across Qwen, Llama, and Olmo, the authors report, with an AUC as high as 0.865 in the gradient space of misaligned models. The researchers also identified tool-execution failures and narrative arithmetic as under-studied sources of safety degradation, which suggests safety work has been looking in only part of the data. m:…1994

The fourth preprint is a technical report that models the same honesty about training. Metis-1.6 consists of two language models, Praxis at 2.94 billion parameters and Logos at 10.23 billion, each processing every token twice through the same 20 layers, trained on 1.02 trillion tokens in total. Logos scores 67.50 percent on AIME 2025. The report is unusually frank about failure: the optimizer configured for training never ran, a 1.77 billion parameter n-gram memory was removed because it learned to switch itself off, reinforcement learning on Logos reduced model breadth and led the team to stop at supervised fine-tuning, and a length reward under group-mean normalization punished correct solutions for being long. The models are released under an Apache 2.0 license, with a custom inference engine named Dromos. m:…3767

Evaluation is catching up

Read together, the four preprints describe evaluation catching up with what agents actually get wrong. Benchmarks now measure whether interacting constraints are honored, whether rankings beat trivial baselines, whether training data silently degrades safety, and whether training failures are disclosed rather than smoothed over. That is a healthier regime than headline scores alone, and it is also a more demanding one for builders: a coding agent that passes single-requirement tasks, a research agent that generates plausible ideas, and a model with a strong benchmark score can each still fail the test that matters. The work ahead is less about new capabilities in isolation and more about proving those capabilities survive contact with interacting requirements. m:…5539 m:…6991 m:…1994 m:…3767