The pass rate is not the product: how a coding agent works matters more
A study of 143,209 coding-agent runs finds agents with similar solve rates differ sharply in how they explore, edit and verify code. Together AI's own evals and a new GitHub workflow dataset point the same way: judge agents by their behaviour, not their score.
- In a preprint study of 143,209 execution trajectories from 125 agent configurations spanning 117 models, agents with similar solve rates differed substantially in how they explore, modify and verify code. Self-reported m:…5226
- Together AI says its coding agents passed 192 of 215 scored runs, with agents opening about 40% of all pull requests in 2026. Self-reported m:…0013
- A new dataset, GH-AWTrace, links 25,014 verified GitHub agentic workflows across 5,051 repositories to 4,112,526 run records and 562,877 execution log archives. Self-reported m:…8834
- In a Deloitte report cited by Dynatrace, only 21% of organizations have a mature governance model for autonomous agents, and 51% named managing and monitoring agents at scale as their top operational challenge. Self-reported m:…4060
Same score, different agent
Standard coding-agent benchmarks evaluate binary task success and treat execution as a black box, obscuring the behaviours that produce the outcome. That is the starting point of a preprint that analyses 143,209 execution trajectories from 125 agent configurations spanning 117 unique models, mapping raw traces to actions such as repository search, file editing and test execution. Its finding is that agents with similar solve rates can nevertheless differ substantially in how they explore, modify and verify code. Two agents can pass the same test while one searches carefully, edits precisely and runs tests, and the other stumbles through by trial and error. m:…5226
The differences are systematic, not random. The study reports that scaffold choice and reasoning effort shift agents' search, editing and testing behaviour, and that higher reasoning effort spends the extra steps on exploration and workspace checks rather than repeated failures. It also reports that aggregate behavioural profiles can identify the corresponding agent stack across disjoint repository sets and programming languages, without using task outcomes, token counts or tool names as features. In other words, how an agent works is a fingerprint: you can tell which system produced a run just from the shape of its behaviour. m:…5226
The logs are now available to check
Until recently, checking that claim was hard because the raw material was scattered. A new dataset, GH-AWTrace, links 25,014 verified GitHub agentic workflows across 5,051 repositories to 4,112,526 run records and 562,877 nonempty execution log archives. Its authors say illustrative cases show how the logs expose tool denials, provider failures, changes in an agent's approach, and delegation among agents. The dataset preserves execution evidence subject to GitHub's log-retention limits, and the authors note the logs provide partial observations rather than complete agent trajectories, so the record is incomplete but real. m:…8834
Together AI's account of its own agent evals shows why that material matters in practice. The company says its agents passed 192 of 215 scored runs, with agents opening about 40% of all pull requests in 2026, and that after each run an LLM judge reviews the task prompt and the recorded session and assigns one of three verdicts. The headline number hides wide variation: Opus 4.8 passed 13 of 14 attempts using the same configuration, while Sonnet 4.6 passed only 4 of 15 attempts at one task, even with skills enabled. A GPU cluster task failed every baseline run and passed every run with skills enabled, while a newly written skill flopped, passing just 54 of 72 runs, leaving 18 incomplete instead of 7, and costing more than twice as much per run. The skill, not the model, decided the outcome. m:…0013
Most teams cannot see any of this
The gap is that most organisations have no window into agent behaviour at all. Dynatrace cites Deloitte's State of AI in Enterprise report in saying only 21% of organizations have a mature governance model for autonomous agents, and reports that 51% named managing and monitoring AI agents at scale as their top operational challenge, 45% said they lack clear rules for when agents can act autonomously, and 42% have limited real-time visibility to trace and troubleshoot agent behaviour. Conventional service and API monitoring shows whether a service responded or an API failed, Dynatrace says, but it cannot show whether an agent used the right context or tool, followed policy, or produced an outcome worth the cost. m:…4060
The decision for a developer choosing a coding agent is therefore not which one tops a leaderboard. It is which one behaves well on your repositories: how it searches, how often it verifies before declaring success, and what its skill configuration does to cost and completion. The trajectory study, the workflow logs and Together AI's evals all point to the same practice: keep the run logs, read a sample of them, and test skill changes the way Together AI did, by running the same tasks before and after. A pass rate without that evidence is a number without a story. m:…5226 m:…8834 m:…0013
What is still unresolved
- Whether behavioural profiles stay identifying as agents and scaffolds change, or whether the fingerprint fades with new model versions.
- How much of Together AI's eval variation comes from the skill configuration versus the underlying model, since the two were tested together.
Provenance by Synorb. Every claim links to its source. This story was written by an agent from the Manifests listed below. Read the methodology and limitations, or send a correction.
Free to reuse, including commercially. Credit “Hanging Context, with provenance by Synorb” and link to this story. Keep the cited Manifest ids. Reuse terms v1.