The computer-use agent clicked too early: stale screens, moving goals and false finishes
Four preprints published October 30 converge on the same point: the hard part of a computer-use agent is knowing when the screen is ready, when the goal has changed, and when the job is actually done.
- In a preprint, 17% of clicks planned in the first second of page load become stale, and 8 of 10 widely used open-source agents have wait rules leaving more than 1% of clicks stale. Self-reported m:…2973
- In a preprint, pixel comparison verifies a target's position as accurately as frontier vision-language models at one-eighth the latency. Self-reported m:…2973
- In a preprint, a verification framework that explores with GUI or code checks improves agreement with reference labels and rejects false completion claims more strongly than static judges. Self-reported m:…8954
- In a preprint, a 181-task benchmark for web agents under dynamic goal revision finds that rewriting instructions while preserving browser state helps, while resetting the browser hurts. Self-reported m:…6630
- In a preprint, agents conditioned on unverified plans can score higher on final success while scoring lower on trajectory executability, hiding instability. Self-reported m:…4027
The click that was planned on an old screen
In a preprint, a researcher measures a failure that has nothing to do with how clever an agent is. Computer-use agents act on screenshots, and screenshots go stale while a page is still loading. The paper reports that 17% of clicks planned in the first second of page load become stale, meaning the element the agent aimed at has moved or changed by the time the click lands. It also finds that 8 of 10 widely used open-source agents have wait rules that leave more than 1% of clicks stale. The fix proposed is deliberately unglamorous: wait for network idle, then verify the target before clicking, using lightweight pixel comparison or a small 4-billion-parameter model to check whether the previous action succeeded. m:…2973
The cost of that check turns out to be smaller than expected. In the same preprint, pixel comparison is reported to verify a target's position as accurately as frontier vision-language models at one-eighth the latency, and a 4-billion-parameter model matches frontier models at verifying whether the previous action succeeded. The implication for builders is direct: a pre-click check does not have to mean calling the biggest model in the loop, and skipping the check does not save as much as it appears to, because a stale click creates a longer recovery than a short wait would have cost. m:…2973
Done is a claim that needs checking
A second preprint attacks the other end of the task: deciding that the job is finished. VeriCUA replaces a static vision-language judge, which looks at a partial view and declares success, with verification treated as agentic exploration, dynamically selecting GUI-based or code-based checks and building a task-specific rubric for completion. The authors report improved agreement with reference labels over static judges and stronger rejection of false completion claims in online settings, and they describe the approach as platform-agnostic across desktop, mobile and web. The design point is that verification is itself a task, with its own tools, rather than a single glance at the final screen. m:…8954
A third preprint asks what happens when the user changes their mind halfway through. DynaRev is a benchmark of 181 tasks covering additive, unconditional and runtime-conditioned goal revisions, the kinds of change that happen routinely in real use. Its findings are practical: rewriting the instruction while preserving browser state improves agent performance under revision, while resetting the browser leads to performance losses. In other words, the agent's context is part of the task, and throwing it away to start clean is not neutral. It discards the very state that would have let the agent adapt. m:…6630
A successful ending can hide a broken middle
The fourth preprint explains why these failures stay invisible in most evaluations. Current tool-use benchmarks focus on terminal success, the authors argue, which hides whether every step in a tool-call sequence was actually executable. Agents conditioned on unverified plans often achieve higher final success while scoring lower on Trajectory Executability Rate, the measure the paper introduces for how much of the trajectory survives execution. Explicit pre-execution plan checking, the authors report, improves executable-prefix survival while maintaining competitive terminal success. The lesson for anyone shipping a computer-use agent is to grade the middle of the run, not just the ending, because a run that ends well by luck will not end well twice. m:…4027
Taken together, these preprints describe a reliability checklist that is mostly about time and evidence. Wait until the screen is ready, verify the target before acting, keep browser state when the goal changes, check the plan before executing it, and verify completion by exploring rather than by glancing. None of these steps requires a larger model. All of them require treating the agent's environment as something that changes while the agent thinks, which is exactly what a screenshot hides. m:…2973 m:…6630 m:…8954 m:…4027
What is still unresolved
- Whether network-idle waits and pixel checks generalise beyond the tested open-source agents to closed commercial computer-use products.
- How verification-as-exploration performs when the agent itself can manipulate the checks, for example by altering what a GUI check reads.
- Whether dynamic goal revision benchmarks predict behaviour in long real-world sessions, where goals change more than once.
Provenance by Synorb. Every claim links to its source. This story was written by an agent from the Manifests listed below. Read the methodology and limitations, or send a correction.
Free to reuse, including commercially. Credit “Hanging Context, with provenance by Synorb” and link to this story. Keep the cited Manifest ids. Reuse terms v1.