GUI agents learn what to remember and how to look

Remember the right thing, the right way

GUI agents have a memory problem, and it is not capacity — it is format. In a preprint, the SwitchMem authors show that whether an agent should remember a step as text or as an image depends on the step itself, which quietly indicts the current default of picking one representation and sticking with it. Their router, SwitchMem, reads preference signals from the actor's hidden representations and dynamically selects between textual and visual memory. They report it beating static memory-augmented baselines across three GUI benchmarks. m:…7878

The practical implication is small and immediate: if your GUI agent keeps one memory format, you are leaving performance on the table at the steps where the other format would have worked. Adaptive memory is a routing problem, not a model problem, which makes it one of the cheaper upgrades on this list. m:…7878

The finding lands harder against the status quo. In a preprint, the authors note that existing memory-augmented GUI agents typically use a fixed representation strategy — pick text or vision, commit, done. SwitchMem reads preference signals from the actor's hidden representations to choose per step instead, and the authors have released code and project resources alongside the paper. The upgrade path is unusually clear for a research result: a router, not a retrain. m:…7878

Stop rendering the future

The Latent World Action Model paper makes a delightfully contrarian argument: GUI world models waste enormous compute drawing pictures nobody needs. Pixel-space world models predict full future screens, then re-perceive them to choose actions — reconstructing appearance-level detail far beyond what action selection requires, at high computational cost, and occasionally falling into an 'identity shortcut' where the generation objective goes slack. The authors move the prediction target into latent space: predict only the action-induced future information. m:…1709

In a preprint, they report up to 31.9% relative success gains in offline GUI settings, 40.6% online, and — the number that should make infrastructure teams sit up — up to 98.9% lower inference latency from dropping pixel generation. If you run GUI agents at any scale, latency is your unit economics; a 50x-class cut from an architectural change rather than a hardware upgrade is the kind of result that re-prices a roadmap. m:…1709

The architecture is more specific than simply predicting less. In a preprint, the authors describe a large multimodal model serving as the latent world model, with a transition head capturing action-conditioned future information and an action head translating latent representations into action selection — plus an action-aware contrastive objective to mitigate the identity shortcut, where the generation objective would otherwise go slack. It is a full redesign of what a GUI world model is for: not rendering futures, but choosing actions. m:…1709

Evolve how the agent looks

VIEWCRAFT completes the trio by evolving not the model but the evidence interface — how observations get selected, transformed and organized before the agent reads them — jointly with the reading procedure itself, and without any weight updates. In a preprint, the authors report average gains of 6.1 points across eight benchmarks, with learned interfaces using 8-15% fewer input tokens on multi-source tasks than the starting skills, and skills that transfer across different executors. The takeaway: the interface is trainable too. m:…1873

Across all three preprints, the theme is that GUI agent progress is coming from the plumbing around the model — memory routing, prediction targets, evidence interfaces — rather than bigger vision-language backbones. For developers, that is good news: plumbing is cheaper to change than models, and the reported wins (latency down 98.9%, tokens down 8-15%, success up double digits) are the kind that show up directly in the bill. As always with preprints, replicate before you commit. m:…1709 m:…1873 m:…7878

One more detail worth pocketing: VIEWCRAFT's learned skills transfer across different executors and beat direct execution in every transfer scenario the authors tested, per the preprint. That means the interface work is portable — you can evolve the evidence pipeline once and reuse it as you swap underlying models. In a year where everyone is changing backbones quarterly, portable plumbing is the durable investment. m:…1873

The training loop has craft in it. In a preprint, the authors describe failure traces guiding revisions, with replay and paired evaluation retaining only useful changes, and interface projection recovering visual edits from eligible rejected joint proposals. They also report the performance advantage persists at a matched input-token budget — the gains are from better evidence construction, not from spending more tokens. That is the result that separates a real method from a bigger prompt. m:…1873