Computer-use agents merge clicks with code — then stumble on real users

Clicks meet code on the desktop

The desktop agent stack took a step this week with OS-Helix, described in a preprint as an end-to-end trained multimodal agent that integrates GUI actions and code execution through hybrid supervised and reinforcement learning. The headline number: OS-Helix-27B-RL reached 66.44% success on OSWorld-Verified, the authors’ headline result, and 63.87% on WindowsAgentArena with no Windows-specific training data at all. Getting there was not trivial: naively adding a code tool to a GUI agent degrades performance, the authors report, and their step-balanced GSPO training fixes the imbalance. m:…8518

The cautionary finding is just as useful for builders: naively bolting a code tool onto a GUI agent degrades performance. The fix, step-balanced GSPO, aggregates importance weights by action step rather than raw assistant tokens, so long code actions cannot dominate optimization. Merging clicks and code is the future; doing it naively makes things worse. m:…8518

Phones get the open stack

OpenMobile-2 is the mobile counterpart — a near-frontier mobile agent with fully open training environments and recipes, described in a preprint. The scale is serious: a playground spanning over 110 apps, over 300 app-native tools across 50+ apps, nearly 12K curated mobile interaction trajectories for supervised fine-tuning, and over 2K executable tasks with automatic rewards for reinforcement learning. m:…3069

Results: SPA-Bench performance nearly doubled from 31.9 to 59.6, with 79.9 on AndroidWorld and 50.4 on MobileWorld. The whole stack — environments, tools, trajectories, rewards — is open, which is the part developers should care about most. m:…3069

Read the manual, not the demo

MAGE asks what happens when the agent meets software it has never seen. The answer, in a preprint: feed it the user manual. The framework converts manuals into executable task-plan pairs for environment rollouts, cutting reliance on human-authored training data, and adapts the policy through environment interaction to handle discrepancies between the manual and the actual GUI. Documentation — the most boring artifact in software — becomes training data. m:…0053

The 36% robustness cliff

Then the cold water. PersonaCUABench, in a preprint, rewrites task instructions through diverse personas instead of fixed specifications and finds task completion drops 36%. User satisfaction falls from 84.1% to 62.7%. The headline capability numbers above were measured on clean, fixed instructions; real users do not write like that. m:…8814

And these are not refusals — 59.9% of failures are agent errors despite the agent engaging with the request. My reasoning: this is the deployment gap, quantified. Agents work for the prompt engineer and stumble on the actual user. m:…8814

Cheaper to run, safer to correct

Two more preprints attack deployment cost and control. ECHOES shows GUI agents can reach competitive performance without retaining historical screenshots at deployment — dropping a real memory bill — via recursive Compression-Think-Action updates that keep task state in text. Its PMF Gap-guided OPD-RL dynamically balances privileged distillation against task-reward optimization during training. m:…1130

ECHOES also ships ECHOESBMK, a fine-grained diagnostic benchmark for GUI agent failure trajectories — the tooling to see where these agents break, not just whether they do. m:…1130

Structured Authority Injection tackles the human-in-the-loop problem: instead of free-text corrections that decay over a session, it routes corrections through typed, schema-validated MCP tool calls with three override classes — Constraint, Priority, Strategy — each with its own persistence semantics and precedence order. The authors report up to 38% lower prompt overhead in strong models. m:…2261