No code ships until the evals pass: the gate moves before production

Define good before you build

LangChain's answer to non-deterministic agents is to start with the test. Eval-driven development means building an agent while measuring it against a consistent set of metrics and climbing toward the eval, the company explains. The first step is a short spec of what good looks like — the behavior the agent should show — followed by a dataset of inputs and a way to score the runs. The loop is evaluate, compare, adjust, repeated until the numbers move; the company calls this hill-climbing. m:…4294

The reason the order matters is that agents are not ordinary code. Outputs vary, more than one answer can be correct, and the agent chooses its own steps, so a fixed unit test cannot cover the behavior. Evals are more flexible than traditional tests, LangChain says, because any aspect of a run can be scored and another language model can do the scoring. Real agent runs are one of the best sources for dataset inputs, because users phrase things in ways developers would not predict. And the company warns about the trap: keep adjusting the agent until every example passes and it starts overfitting to the dataset, passing the eval while missing requests phrased in ways the examples do not cover. Automated evaluation in CI/CD and production, the company says, keeps untested changes from reaching users and catches regressions before production — and with LangSmith integrating with pytest and Vitest, evals can run on every pull request. m:…4294

The gate at one company

Packsize, which builds on-demand packaging systems, shows what the gate looks like in practice. AI coding tools increased the volume of pull requests until the bottleneck moved from writing code to reviewing and verifying it, the company says, and its answer was to make Qodo review mandatory: no code goes to production without Qodo taking a look first. The company's AI architect, Nathan Hansen, says the team quickly found that the AI that wrote the code was generally fine with its own implementation — so Qodo reviews with several models working adversarially, and pulls review standards automatically from skill.md files in each repository. Across roughly 50 microservices, review time now takes just minutes, the company says, and Qodo's review once flagged a compatibility fix in shared packages that would otherwise have caused a production outage. m:…8557

Greptile is rebuilding the review surface itself. Its new PR inbox puts CI checks, merge conflicts, required approvals and the Greptile review in one place when a pull request is opened, and a code tour tab groups related edits and explains what each change does alongside the surrounding code. When Greptile finds an issue, the developer can send it straight to a coding agent to fix, and an integrated chat answers questions using context from the review and the selected code. Bulk approval and merging, plus keyboard shortcuts, let reviewers work through a queue. The two products attack different halves of the same problem: Greptile speeds up the human review, Qodo makes an independent review the condition for shipping. m:…7563

Benchmarks that test real work

The same gate logic is showing up in benchmarks that test whether models can do real work, not just answer questions. Anthropic says Claude Fable 5.2 can now design and prototype electronic hardware from start to finish, and the company tested it with Diode Computers, a firm that designs and manufactures circuit boards. The model designed three devices — a speakerphone, a voice-assistant hub and a handheld virtual pet — wrote its own router algorithm to lay out long tracks on the voice-assistant hub board, and ran design-for-manufacturing and sourcing checks on its own designs. It scored 69% on the PCB Gym benchmark, against 25% for Fable 5, and 83.5% on BenchCAD 2.1, nearly double its predecessor's score. Diode manufactured all three boards directly from Claude's design files, without adjustments. The prototypes have not yet been through electrical validation, antenna tuning, thermal testing or certification, Anthropic says — the gate still has stages the model has not passed. m:…7585

Frontier3DBench, a preprint benchmark of 145 tasks across nine categories, tests coding agents on 3D creation across objects, scenes and interactive open worlds. Eleven model-harness configurations were tested. GPT-6.1-sol led the object and scene levels with the highest overall score of 66.02 out of 100, while GPT-6-astra led interactive world construction. The pattern the authors highlight is a split: every configuration scored lower on reference reconstruction than on text-driven generation, and generated worlds averaged 83.08 out of 100 on local mechanisms but only 20.82 on motion clearance and 30.26 on stateful workflows. Reference fidelity and coordinated activity completion, the authors say, are the central bottlenecks — individual mechanisms work, but complete activities that need coordinated actions and persistent state remain difficult. m:…0736

What changes for teams

The thread running through all five is that the check now comes before the ship decision. A spec and a dataset before the code, an independent review before production, a benchmark that scores finished artifacts rather than chat replies. The implication for a development team is a shift in where time goes: less of it writing the first draft, more of it defining what good looks like and building the gate that enforces it. The teams that do that work early get faster reviews and fewer outages; the ones that skip it get the same volume of AI-written code with nowhere for it to be checked. m:…4294 m:…8557 m:…7585