The Harness Is the Product: Why Agent Design Starts Outside the Model

A model learns its harness

In a preprint, researchers argue that language-model agents learn preferences specific to the prompts and tools used during training, a phenomenon the authors call harness preferences. Under KL-regularized control, a policy encodes its harness preferences, and deployed performance equals harness potential minus policy mismatch. Changing a harness moves both terms, which means comparing harnesses with a fixed model mixes what each harness offers with how well that model uses it. For coding-agent harnesses on Terminal-Bench Lite, the authors report that ceiling minus mismatch reproduces the accuracy ranking, a result that gives the decomposition empirical weight rather than leaving it as a slogan. m:…4192

The most vivid demonstration in the preprint is a small change with a large effect. In MiniWoB++, reversing the order of observation fields lowered a trained model success rate from 85.0 percent to 19.2 percent. That is not a story about a weaker model; it is a story about a policy that had learned to rely on the layout of its interface. The preprint also reports that new harnesses that raise the ceiling outperform the original agent after further training, and that ceiling and mismatch measured beforehand predict performance gains. The implication for builders is direct: a harness change is a retraining event, not a configuration tweak, and its payoff should be forecast before it is shipped. m:…4192

Verification that pays

In a preprint, researchers evaluated 17 verification designs for agent harnesses across seven frontier coding models using Terminal-Bench, SWE-bench Verified, and VisualBench. Independent critic review improved average accuracy by 5.9 to 9.5 percentage points on Terminal-Bench and by 3.9 to 6.6 percentage points on SWE-bench Verified, while costing 1.13 to 1.25 times the baseline cost on Terminal-Bench. Independent review also achieved net preferences of 13.2 to 23.8 points over the baseline on VisualBench. Completion gates could yield larger accuracy gains but might substantially increase cost, while increasing review rounds or the number of critics improved average Terminal-Bench accuracy by up to 3.5 percentage points. m:…1468

Initial instructions and execution-time reminders yielded only modest, model-dependent average gains on coding benchmarks in the same study, which sharpens the contrast. The cheapest reliable win the researchers found was not more prompting but an independent second look, structured as a critic review rather than as a gate that blocks completion. That finding fits the harness-geometry argument: verification is part of the harness, and its design changes both the ceiling of what the system can catch and the cost of catching it, so it should be chosen with the same care as the model itself. m:…1468

Learning on both levels

In a preprint, researchers introduce HaMo, a framework for long-horizon continual learning in web agents that works on two levels at once. The harness level distills trajectories into compact experience cards stored in persistent non-parametric memory for immediate retrieval, and a retrieval-coverage-based selection mechanism identifies trajectories for parameter updates. The model level updates a parameter-efficient adapter using selected trajectories and historical replay, under an experience-conditioned continual learning objective. HaMo combines non-parametric experience accumulation with parametric model adaptation, and the authors report that it outperforms harness-only adaptation and model-only adaptation in within-domain and cross-domain shifts. m:…9564

HaMo also improved retention of previously acquired capabilities and adaptation to new tasks, according to the preprint. That combination matters because long-horizon agents face two failure modes at once: forgetting what they knew, and failing to transfer what they learned to a new task. A harness-only system can retrieve old experience but cannot internalise it, while a model-only system can internalise but cannot retrieve on demand. HaMo treats the two as complements, and its reported gains on both retention and adaptation suggest the split is where the engineering leverage sits. m:…9564

Design rules that follow

Read together, the three preprints give agent teams a short set of design rules. First, measure harness potential and policy mismatch separately before changing an interface, because a change that raises the ceiling can still hurt if it increases mismatch more. Second, put an independent critic review into the harness before adding more prompting, because the verification study shows it is the most favourable accuracy-cost trade available in that comparison. Third, for agents that must keep learning, split the work between retrievable experience cards and adapter updates, rather than betting on one level alone. Each rule follows from a measured result, not from a preference for one architecture over another. m:…4192 m:…1468 m:…9564