The agent reliability stack arrived all at once: evals, tripwires, and an owner for every agent
LangChain, Vercel, Google, and AMD with Fiddler all shipped tools to watch agents in production. The shared pitch: an agent without an eval and an owner is a liability.
- LangChain rebuilt LangSmith around the agent as the core object, with each agent record carrying an owner, a globally unique immutable ID, and a type. Self-reported m:…7682
- Vercel says end-to-end evals are the most effective proxy for the effort, time, and cost of agentic testing, and that trace analysis is necessary to understand why agents fail. Self-reported m:…4823
- Google's Gemini Agent Platform supports Agent Anomaly Detection, which monitors agent telemetry and catches behavioural risks. Self-reported m:…5537
- AMD and Fiddler's partnership aims to provide visibility, governance, and predictable costs for enterprise agentic AI, with guardrails that screen prompts and responses for jailbreaks, toxicity, and personal or health data. Self-reported m:…3922
What shipped
LangChain has rebuilt LangSmith around a single object: the agent. The company says its new LangSmith Agent Platform is one place to build, run, improve, and govern agents, and that agents are now the core object across LangSmith, connecting workflows that used to sit apart. Each agent record carries an owner, a globally unique immutable ID, and a type. The platform offers governance for agents running on any platform that supports the OpenTelemetry Protocol, and it is available globally in cloud, bring-your-own-cloud, and self-hosted environments. It supports general purpose and coding agent types today, with voice agents planned. m:…7682
The point of the rebuild is accountability. An agent that used to be a tangle of prompts, tools, and traces now has a name, an owner, and an ID that does not change. That is a small design decision with large consequences: once an agent has an owner and an ID, its failures belong to someone, and its behaviour can be tracked across runs instead of disappearing into a log file. m:…7682
Evals are the unit of work
Vercel's argument starts one step earlier, with how you test. The company says agents follow the same product funnel as humans: discovery, selection, access, and success. Its testing pyramid groups tests by how many language-model calls they need: deterministic tests need none, one-turn tests need one, and end-to-end tests need two or more. End-to-end evals, Vercel says, are the most effective proxy for the effort, time, and cost of agentic testing. Two measures matter: Pass@k, the probability that at least one of k trials succeeds, and Pass^k, the probability that all k trials succeed. m:…4823
Scores alone are not enough, Vercel says, because a score tells you how often an agent fails, not why. Trace analysis is necessary to understand why agents fail, and automated trace analysis can use reader agents to summarise trial transcripts. The company also ships Harbor, an open-source framework that runs agents in containers, checks results, and records actions. One warning comes with the framework: evals saturate when models and products improve, meaning agents pass every task. An eval suite that never fails is not proof of reliability; it is proof the test is too easy. Vercel's AI Gateway, meanwhile, routes model calls through one key and one budget across providers. m:…4823
The tripwire layer
Google's contribution is a guide to catching an agent that has gone wrong while it is still running. The company's Gemini Agent Platform supports Agent Anomaly Detection, which monitors agent telemetry and catches behavioural risks. The guide walks through deploying a sample customer support agent, configuring proactive alerts through Cloud Monitoring and Security Command Center, and triggering out-of-scope security findings. The idea is simple: an agent's behaviour has a shape, and a tripwire fires when the shape changes. m:…5537
AMD and Fiddler make the same point from the infrastructure side. Their partnership aims to provide visibility, governance, and predictable costs for enterprise agentic AI. The problem they describe is concrete: healthy infrastructure metrics can hide incorrect AI answers, a single agentic request can involve retrieval, multiple model calls, and tool actions, and traditional monitoring cannot pinpoint which span in a trace of a thousand caused a bad outcome. Fiddler's answer is a control plane that records agentic systems from the application down to individual spans, guardrails that screen prompts and responses for jailbreaks, toxicity, and personal or health data, and models that run inside the customer's environment without external API calls. The company also keeps a unified agent registry for every agent, model, and application. Adversarial prompts can push an agent outside its intended boundaries, Fiddler says, which is why evaluating requests inside the customer's own environment matters: it catches rare jailbreaks and policy violations that a lab test would miss. m:…3922
What this changes
Four vendors, four products, one diagnosis. The agent is no longer a script you run; it is a system you operate, and operating it needs the same machinery any production system has: an owner, a test suite, a registry, and an alarm. The open question is whether teams will actually gate deployments on evals, or whether the evals become a dashboard nobody watches. Another is whether anomaly detection catches failures nobody has seen before, rather than the ones the guide already describes. And the cost question is unanswered: evals, traces, guardrails, and registries all consume the same tokens and compute they are meant to police. m:…7682 m:…4823 m:…5537 m:…3922
What is still unresolved
- Whether teams gate agent deployments on eval scores, or treat evals as a dashboard.
- Whether anomaly detection catches novel failure modes, not just the ones in the guide.
- What the full eval, trace, and guardrail stack costs to run against the agents it watches.
Provenance by Synorb. Every claim links to its source. This story was written by an agent from the Manifests listed below. Read the methodology and limitations, or send a correction.
Free to reuse, including commercially. Credit “Hanging Context, with provenance by Synorb” and link to this story. Keep the cited Manifest ids. Reuse terms v1.