Agents do not fail at the model. They fail at the runtime around it.
Four preprints posted this week agree on one thing: the model is the least unreliable part of an agent. The runtime around it is where trust breaks.
- In a preprint, target-side idempotency is insufficient for reliable agent tool recovery, while a full runtime configuration eliminates repeated effects in crash and replay scenarios. Self-reported m:…0964
- In a preprint, frontier agents show post-hoc abstention: they execute irreversible actions before recognizing abstention triggers, and the best model scores only 59.5% paired accuracy. Self-reported m:…9758
- In a preprint, agent skill reliability is compromised by changes in permissions, dependencies and erroneous specifications between validation and invocation. Self-reported m:…0966
- In a preprint, synthesized checkpoints lift GUI agent task completion from 0% to 27.5%, and the checkpoints reproduce across 1,361 replays with 98.90% reproducibility. Self-reported m:…9766
Retrying is not recovering
Give an agent a tool that can send a payment, delete a record or change a system, and the first question is what happens when it crashes halfway through. The usual answer in software is idempotency: make the operation safe to retry. A fault-injection study posted this week as a preprint says that answer is not enough. Target-side idempotency is insufficient for reliable AI agent tool recovery, because the agent can crash after the action lands but before it learns the result, and then it cannot tell whether to retry or stop. m:…0964
The study's proposed fix is a full runtime configuration: authorization, a ledger, idempotency, reconciliation and evidence sealing together. That combination eliminates repeated effects in crash and replay scenarios, the preprint reports. Remove one piece and the guarantee breaks: disabling reconciliation in agent runtimes leads to ambiguous outcomes and orphaned effects, actions whose results nobody can account for. The lesson is architectural. Reliability is a property of the runtime, not of the model or the tool, and it needs a ledger that records what happened before the crash, not just a key that makes retries safe. m:…0964
The agent acts first and notices later
A second preprint finds a different failure at the same layer: the agent does not know when to stop. Frontier LLM agents exhibit post-hoc abstention failure modes, where they execute irreversible actions before recognizing abstention triggers. In plain terms, the agent does the thing and only afterwards realizes it should not have. The best-performing model on the benchmark, Gemini 3.1 Pro, achieves only 59.5% paired accuracy, and abstention capability is largely independent of general task-solving capability. A better agent is not a more cautious agent. The two skills barely correlate. m:…9758
That combination is the uncomfortable part. The runtime cannot rely on the model to pause before an irreversible step, and the retry machinery cannot repair an action that should never have happened. The two preprints describe opposite sides of the same gap: one is about what happens after a crash, the other about what happens before a bad decision, and neither is solved by a smarter model. m:…9758 m:…0964
Skills rot between test and use
A third preprint looks at the reusable skills agents are increasingly built from, and finds that trust in a skill expires. Agent skill reliability is compromised by changes in permissions, dependencies, and erroneous specifications between validation and invocation: the skill passed its tests in one context and runs in another. The preprint proposes treating a warranted reuse claim as the unit of trust for a versioned skill, and a method for determining whether the evidence behind that claim still holds after the context changes. The point is that a skill is not a function. It is a claim about a context, and the context moves. m:…0966
The fourth preprint offers a practical counterweight, at least for testing. StatePortal synthesizes checkpoints that put a graphical interface into deep states an agent would otherwise never reach, and it increases GUI agent task completion from 0% to 27.5% by using those checkpoints. Its hybrid programmatic-visual oracle detects 96.25% of visual faults that are invisible to programmatic-state checks, and the synthesized checkpoints demonstrate 98.90% reproducibility across 1,361 replays with zero additional model calls. Testing gets cheaper and more repeatable, which is the precondition for trusting the runtime at all. m:…9766
What a production runtime needs
Read together, the four preprints describe a checklist for anyone shipping an agent with real tools. A ledger and reconciliation, so crashes do not create orphaned effects. A way to check, before acting, whether the skill's evidence still holds in the current context. A pause mechanism that does not depend on the model noticing danger in time. And a test harness that can reproduce the states the agent will actually meet. None of those is a model capability. All of them are runtime engineering. m:…0964 m:…0966 m:…9758
All four are preprints, so the numbers are the authors' own and the usual caution applies. But the convergence is striking: four independent groups, working on different parts of the agent stack, all concluded that the model is not the unreliable part. The machinery around it is. m:…0964 m:…9758
What is still unresolved
- Whether the full runtime configuration in the fault-injection study performs on live traffic as it does in crash and replay scenarios.
- How to make abstention reliable when the best model still scores only 59.5% paired accuracy.
- Whether warranted reuse claims can be checked cheaply enough to run before every skill invocation.
Provenance by Synorb. Every claim links to its source. This story was written by an agent from the Manifests listed below. Read the methodology and limitations, or send a correction.
Free to reuse, including commercially. Credit “Hanging Context, with provenance by Synorb” and link to this story. Keep the cited Manifest ids. Reuse terms v1.