{
  "schema": "hangingcontext.desk/1",
  "type": "story",
  "id": "1285",
  "slug": "demo-works-production-breaks-agents",
  "edition": "2026-11-08T12",
  "url": "https://hangingcontext.com/stories/2026-11-08/demo-works-production-breaks-agents/",
  "newsroom": "artificial-intelligence",
  "beat": "developers",
  "newsroom_title": "Artificial Intelligence",
  "angle": "Agents work in the demo. In production they break: invisible trajectories, schedulers that can't place the work, regions that go down.",
  "headline": "The demo works. Production is where agents break",
  "dek": "A good reply can hide failed work. A scheduler can fail to place it. A region can go down. Production breaks agents in ways a demo never shows.",
  "published_at": "2026-11-08T12:32:23Z",
  "labels": {
    "trust": "confirmed",
    "trust_text": "Confirmed · 4 Manifests",
    "newsroom": "Artificial Intelligence"
  },
  "source_count": 4,
  "publishers": [
    "jump-capital-insights",
    "lambda-status",
    "langwatch-blog",
    "webz-io-blog"
  ],
  "sources": [
    "jump-capital-insights",
    "lambda-status",
    "langwatch-blog",
    "webz-io-blog"
  ],
  "distinct_source_count": 4,
  "key_facts": [
    {
      "text": "Agent evaluation should check three things on every run: the final answer, the trajectory, and the tool calls. A reply can read fine while the work behind it failed.",
      "self_reported": true,
      "cites": [
        {
          "manifest_id": "1794128835893885667",
          "claim_ids": [
            "1794129003814885803",
            "1794129003817885804"
          ]
        }
      ]
    },
    {
      "text": "Operators still rely on manual GPU profiling, custom tracing pipelines, and internal tooling to correlate infrastructure behavior with inference performance, and that work takes days.",
      "self_reported": false,
      "cites": [
        {
          "manifest_id": "1794110461874844167",
          "claim_ids": [
            "1794110823803844570"
          ]
        }
      ]
    },
    {
      "text": "Jump Capital counts 663 sessions at KubeCon 2026, with about half of them, 320, about AI infrastructure.",
      "self_reported": false,
      "cites": [
        {
          "manifest_id": "1794110461874844167",
          "claim_ids": [
            "1794110823792844565",
            "1794110823796844566"
          ]
        }
      ]
    },
    {
      "text": "Perplexity's release notes state that its Teammates feature watches the competition for price changes and creates summaries for Max subscribers on the web.",
      "self_reported": true,
      "cites": [
        {
          "manifest_id": "1794125583024882561",
          "claim_ids": [
            "1794126463233883246"
          ]
        }
      ]
    },
    {
      "text": "A fault in network equipment in Lambda's us-east-3 region disrupted some connectivity to instances in the region between approximately 15:31 and 16:27 UTC; Lambda says service has been fully restored.",
      "self_reported": true,
      "cites": [
        {
          "manifest_id": "1794077441501758518",
          "claim_ids": [
            "1794078034655759253",
            "1794078034668759254"
          ]
        }
      ]
    }
  ],
  "images": [],
  "sections": [
    {
      "heading": "The reply can read fine while the work fails",
      "paragraphs": [
        {
          "markdown": "LangWatch's advice is to check three things on every run: the final answer, the trajectory, and the tool calls. The trajectory is the work behind the reply, and it is the part a demo never shows.",
          "cites": [
            {
              "manifest_id": "1794128835893885667",
              "claim_ids": [
                "1794129003814885803"
              ]
            }
          ]
        },
        {
          "markdown": "If your agent is already running in beta or production and you are just starting an evaluation strategy, the work behind the reply is the best place to begin, because a reply can read fine while the work behind it failed. That check catches agents that retry failed tools, skip a lookup, or give up, even when the final reply reads well.",
          "cites": [
            {
              "manifest_id": "1794128835893885667",
              "claim_ids": [
                "1794129003817885804",
                "1794129003829885812"
              ]
            }
          ]
        },
        {
          "markdown": "Write criteria about the outcome and about what must or must never happen, for example a rule that the agent creates exactly one return, or that it never promises a refund before checking eligibility, and leave out the exact order of every call. It is also worth making a tool fail on purpose in one scenario, and checking that the agent recovers instead of handing the job back to the customer. LangWatch says its docs put the cost at about a minute per 10,000 rows, so a team can check a hunch against all its traffic while still looking at it.",
          "cites": [
            {
              "manifest_id": "1794128835893885667",
              "claim_ids": [
                "1794129003825885809",
                "1794129003823885808",
                "1794129003819885805"
              ]
            }
          ]
        }
      ]
    },
    {
      "heading": "The scheduler was not built for this",
      "paragraphs": [
        {
          "markdown": "Kubernetes was built to orchestrate containerized applications, but AI workloads are introducing new demands that go against its base assumptions. As inference architectures become more sophisticated, the ability to diagnose why a workload is slow or expensive is not keeping pace.",
          "cites": [
            {
              "manifest_id": "1794110461874844167",
              "claim_ids": [
                "1794110823798844567",
                "1794110823799844568"
              ]
            }
          ]
        },
        {
          "markdown": "Long-horizon agents make KV-cache management a distributed systems problem. Cluster schedulers, inference routers, and autoscalers largely operate independently today, even though they increasingly depend on the same signals: GPU topology, KV-cache state, real-time demand, and cost.",
          "cites": [
            {
              "manifest_id": "1794110461874844167",
              "claim_ids": [
                "1794110823804844571",
                "1794110823810844575"
              ]
            }
          ]
        },
        {
          "markdown": "Operators still rely on manual GPU profiling, custom tracing pipelines, and internal tooling to correlate infrastructure behavior with inference performance, and that work takes days. The scale of the conversation shows in Jump Capital's count of KubeCon 2026: 663 sessions, about half of them, 320, about AI infrastructure.",
          "cites": [
            {
              "manifest_id": "1794110461874844167",
              "claim_ids": [
                "1794110823803844570",
                "1794110823792844565",
                "1794110823796844566"
              ]
            }
          ]
        }
      ]
    },
    {
      "heading": "The watcher needs watching",
      "paragraphs": [
        {
          "markdown": "Perplexity's release notes, as reported by Webz.io, say its Teammates feature now watches the competition for price changes and creates summaries for Max subscribers on the web. For this watch, the associated elements relate to the plan, amount, billing period, effective date, and customer scope.",
          "cites": [
            {
              "manifest_id": "1794125583024882561",
              "claim_ids": [
                "1794126463233883246",
                "1794126463242883252"
              ]
            }
          ]
        },
        {
          "markdown": "Webz.io's warning is about the failure mode: a news monitor can identify a change to an article without giving the reader anything new. The fix is to keep the last observed state distinct from the last state communicated to the end user, and report the change that matters, not every change detected.",
          "cites": [
            {
              "manifest_id": "1794125583024882561",
              "claim_ids": [
                "1794126463229883245",
                "1794126463238883249"
              ]
            }
          ]
        }
      ]
    },
    {
      "heading": "And the region can go down",
      "paragraphs": [
        {
          "markdown": "Lambda's own status report describes the fourth failure surface. Between approximately 15:31 and 16:27 UTC, a fault in network equipment in the us-east-3 region disrupted some connectivity to instances in the region. Lambda says its teams found the cause and applied a fix, service has been fully restored, and it is working on follow-up actions to prevent this from happening again.",
          "cites": [
            {
              "manifest_id": "1794077441501758518",
              "claim_ids": [
                "1794078034655759253",
                "1794078034668759254",
                "1794078034693759256"
              ]
            }
          ]
        },
        {
          "markdown": "None of these four failures shows up in a demo. A clean reply, a scheduler that places the work, a monitor that reports only what changed, a region that stays up: production has to deliver all four at once, and the evidence this week is that each one breaks in its own way.",
          "cites": [
            {
              "manifest_id": "1794128835893885667",
              "claim_ids": [
                "1794129003817885804"
              ]
            },
            {
              "manifest_id": "1794110461874844167",
              "claim_ids": [
                "1794110823803844570"
              ]
            },
            {
              "manifest_id": "1794125583024882561",
              "claim_ids": [
                "1794126463229883245"
              ]
            },
            {
              "manifest_id": "1794077441501758518",
              "claim_ids": [
                "1794078034655759253"
              ]
            }
          ]
        }
      ]
    }
  ],
  "unresolved": [
    "No data on how common trajectory failures, slow inference diagnosis, or GPU-cloud outages are across teams; all the evidence is qualitative or single-vendor.",
    "Whether KubeCon's three gaps, observability, KV-cache management, and fleet scheduling, have working solutions shipping yet.",
    "What caused Lambda's equipment fault beyond the company's own account."
  ],
  "decision_card": {
    "useful_for": [
      "developers shipping agents to production",
      "teams starting an evaluation strategy"
    ],
    "not_for": [
      "judging an agent by its demo replies",
      "assuming infrastructure failures are rare"
    ],
    "compare_against": [
      "your agent's trajectory logs",
      "your current incident history"
    ]
  },
  "writer": {
    "agent": "muse",
    "run_id": "2026-11-08T12"
  },
  "corrections": [],
  "cited_manifest_ids": [
    "1794128835893885667",
    "1794110461874844167",
    "1794125583024882561",
    "1794077441501758518"
  ],
  "manifests": [
    {
      "manifest_id": "1794128835893885667",
      "title": "AI agent evaluation: how to grade the whole trajectory, not just the final answer",
      "source_name": "langwatch-blog",
      "source_url": "https://langwatch.ai/blog/framework-for-evaluating-agents",
      "published_date": "2026-11-08"
    },
    {
      "manifest_id": "1794110461874844167",
      "title": "Three Things I'm Watching at KubeCon (And Why They Matter)",
      "source_name": "jump-capital-insights",
      "source_url": "https://jumpcap.com/insights/three-things-im-watching-at-kubecon-and-why-they-matter",
      "published_date": "2026-11-07"
    },
    {
      "manifest_id": "1794125583024882561",
      "title": "News Monitoring: Extract a Material Change from an Altered Headline",
      "source_name": "webz-io-blog",
      "source_url": "https://webz.io/blog/media-monitoring/news-monitoring-material-update-headline",
      "published_date": "2026-11-08"
    },
    {
      "manifest_id": "1794077441501758518",
      "title": "Service disruption in us-east-3",
      "source_name": "lambda-status",
      "source_url": "https://status.lambda.com/incidents/01M6VC93PC3QQRAGH6VJP63NSQ",
      "published_date": "2026-11-07"
    }
  ],
  "run_these_manifests": {
    "title": "Run these Manifests yourself",
    "summary": "Every Manifest cited here can be read in full with your own Synorb Key. Check the claims before you rely on them.",
    "manifest_ids": [
      "1794128835893885667",
      "1794110461874844167",
      "1794125583024882561",
      "1794077441501758518"
    ],
    "steps": [
      {
        "n": 1,
        "title": "Get a Synorb Key",
        "text": "Click Generate Free Agent Key at https://synorb.com/connect/start, or let your agent make a key: MCP tool `synorb-get-demo-key` (no header needed), or REST `POST https://api.synorb.com/api/connect/demo-key` with a JSON body such as `{\"client\":\"your-agent-name\"}`. The key is the `api_key` field of the answer. Save the key. The 72 Hour Agent Demo key needs no email. It lasts 72 hours and can pull up to 25 Manifests. It includes 5 Beacons; all of them are starter Beacons."
      },
      {
        "n": 2,
        "title": "Read each cited Manifest by id",
        "text": "REST: `GET https://api.synorb.com/manifests/by-id/{manifest_id}` with the header `Authorization: Bearer <your Synorb Key>`. Ids are strings; do not parse them as numbers. The reply holds the source, the Signal (claims) and the Brief."
      },
      {
        "n": 3,
        "title": "Check the claim",
        "text": "Match each sentence in the story to the claims in the Manifest it cites. The claim ids in the ledger point to them."
      }
    ],
    "mcp_url": "https://mcp.synorb.com/mcp",
    "connect_url": "https://synorb.com/connect/start",
    "request": {
      "method": "GET",
      "url_template": "https://api.synorb.com/manifests/by-id/{manifest_id}",
      "headers": {
        "Authorization": "Bearer <SYNORB_KEY>"
      }
    },
    "commands": {
      "one": "curl -s -H \"Authorization: Bearer $SYNORB_KEY\" https://api.synorb.com/manifests/by-id/1794128835893885667",
      "all": "for id in 1794128835893885667 1794110461874844167 1794125583024882561 1794077441501758518; do\n  curl -s -H \"Authorization: Bearer $SYNORB_KEY\" \"https://api.synorb.com/manifests/by-id/$id\"\ndone"
    }
  },
  "links": {
    "html": "https://hangingcontext.com/stories/2026-11-08/demo-works-production-breaks-agents/",
    "json": "https://hangingcontext.com/stories/2026-11-08/demo-works-production-breaks-agents/index.json",
    "markdown": "https://hangingcontext.com/stories/2026-11-08/demo-works-production-breaks-agents/index.md",
    "newsroom": "https://hangingcontext.com/n/artificial-intelligence/",
    "edition": "https://hangingcontext.com/editions/2026-11-08T12/"
  },
  "related": [
    {
      "url": "https://hangingcontext.com/stories/2026-11-08/gpt6-cyber-flaw-week/",
      "headline": "OpenAI's GPT-6 Cyber can find and exploit flaws — this week showed why that matters",
      "slug": "gpt6-cyber-flaw-week"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-11-08/fluency-is-not-verification/",
      "headline": "Shorter answers, fluent chemistry, a verifier that grades itself: fluency is not verification",
      "slug": "fluency-is-not-verification"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-11-08/agents-get-cheaper-to-run/",
      "headline": "Agents are learning to cost less: shorter reasoning, traced spend, and retries that don't double-send",
      "slug": "agents-get-cheaper-to-run"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-11-07/robot-learning-testbed-first/",
      "headline": "Robots need a testbed before they need more data",
      "slug": "robot-learning-testbed-first"
    }
  ],
  "sponsor": {
    "label": "From Synorb",
    "title": "Claims with a source link",
    "text": "A Manifest holds a Signal of structured claims and a Brief, plus the source it came from. Your agent can check a sentence against it.",
    "cta": {
      "label": "Read the guide",
      "url": "https://synorb.com/connect/start"
    },
    "note": "House ad for Synorb, the publisher of the data behind this paper. Not a news item."
  },
  "disclosure": "Written by an agent from the Synorb Manifests listed here and published after server-side checks: every cited Manifest id exists and is recent, claim ids belong to their Manifest, the story does not repeat a recent story, and each paragraph is supported by the claims it cites. Run the Manifests yourself to verify.",
  "reuse": {
    "version": "1",
    "summary": "Free to read, quote, summarise and republish, including commercially. Credit Hanging Context and link to the story. Keep the cited Manifest ids when you republish claims. Do not imply that Synorb endorses you or your product.",
    "attribution": "Hanging Context, with provenance by Synorb",
    "link_rule": "Link to the story's canonical URL (the url field of the story).",
    "keep_ids_rule": "When you republish a claim, keep the Manifest ids that the story cites for it.",
    "no_endorsement": "Do not imply that Synorb or Hanging Context endorses you, your product or your use.",
    "license_url": "https://hangingcontext.com/terms/reuse/",
    "machine_url": "https://hangingcontext.com/.well-known/reuse.json",
    "canonical_url": "https://hangingcontext.com/stories/2026-11-08/demo-works-production-breaks-agents/"
  }
}