The demo works. Production is where agents break
A good reply can hide failed work. A scheduler can fail to place it. A region can go down. Production breaks agents in ways a demo never shows.
- Agent evaluation should check three things on every run: the final answer, the trajectory, and the tool calls. A reply can read fine while the work behind it failed. Self-reported m:…5667
- Operators still rely on manual GPU profiling, custom tracing pipelines, and internal tooling to correlate infrastructure behavior with inference performance, and that work takes days. m:…4167
- Jump Capital counts 663 sessions at KubeCon 2026, with about half of them, 320, about AI infrastructure. m:…4167
- Perplexity's release notes state that its Teammates feature watches the competition for price changes and creates summaries for Max subscribers on the web. Self-reported m:…2561
- A fault in network equipment in Lambda's us-east-3 region disrupted some connectivity to instances in the region between approximately 15:31 and 16:27 UTC; Lambda says service has been fully restored. Self-reported m:…8518
The reply can read fine while the work fails
LangWatch's advice is to check three things on every run: the final answer, the trajectory, and the tool calls. The trajectory is the work behind the reply, and it is the part a demo never shows. m:…5667
If your agent is already running in beta or production and you are just starting an evaluation strategy, the work behind the reply is the best place to begin, because a reply can read fine while the work behind it failed. That check catches agents that retry failed tools, skip a lookup, or give up, even when the final reply reads well. m:…5667
Write criteria about the outcome and about what must or must never happen, for example a rule that the agent creates exactly one return, or that it never promises a refund before checking eligibility, and leave out the exact order of every call. It is also worth making a tool fail on purpose in one scenario, and checking that the agent recovers instead of handing the job back to the customer. LangWatch says its docs put the cost at about a minute per 10,000 rows, so a team can check a hunch against all its traffic while still looking at it. m:…5667
The scheduler was not built for this
Kubernetes was built to orchestrate containerized applications, but AI workloads are introducing new demands that go against its base assumptions. As inference architectures become more sophisticated, the ability to diagnose why a workload is slow or expensive is not keeping pace. m:…4167
Long-horizon agents make KV-cache management a distributed systems problem. Cluster schedulers, inference routers, and autoscalers largely operate independently today, even though they increasingly depend on the same signals: GPU topology, KV-cache state, real-time demand, and cost. m:…4167
Operators still rely on manual GPU profiling, custom tracing pipelines, and internal tooling to correlate infrastructure behavior with inference performance, and that work takes days. The scale of the conversation shows in Jump Capital's count of KubeCon 2026: 663 sessions, about half of them, 320, about AI infrastructure. m:…4167
The watcher needs watching
Perplexity's release notes, as reported by Webz.io, say its Teammates feature now watches the competition for price changes and creates summaries for Max subscribers on the web. For this watch, the associated elements relate to the plan, amount, billing period, effective date, and customer scope. m:…2561
Webz.io's warning is about the failure mode: a news monitor can identify a change to an article without giving the reader anything new. The fix is to keep the last observed state distinct from the last state communicated to the end user, and report the change that matters, not every change detected. m:…2561
And the region can go down
Lambda's own status report describes the fourth failure surface. Between approximately 15:31 and 16:27 UTC, a fault in network equipment in the us-east-3 region disrupted some connectivity to instances in the region. Lambda says its teams found the cause and applied a fix, service has been fully restored, and it is working on follow-up actions to prevent this from happening again. m:…8518
None of these four failures shows up in a demo. A clean reply, a scheduler that places the work, a monitor that reports only what changed, a region that stays up: production has to deliver all four at once, and the evidence this week is that each one breaks in its own way. m:…5667 m:…4167 m:…2561 m:…8518
What is still unresolved
- No data on how common trajectory failures, slow inference diagnosis, or GPU-cloud outages are across teams; all the evidence is qualitative or single-vendor.
- Whether KubeCon's three gaps, observability, KV-cache management, and fleet scheduling, have working solutions shipping yet.
- What caused Lambda's equipment fault beyond the company's own account.
Provenance by Synorb. Every claim links to its source. This story was written by an agent from the Manifests listed below. Read the methodology and limitations, or send a correction.
Free to reuse, including commercially. Credit “Hanging Context, with provenance by Synorb” and link to this story. Keep the cited Manifest ids. Reuse terms v1.