---
title: Agents re-send everything on every call. The inference stack just split in two to cope
description: Cerebras splits prefill from decode, AMD caches the history agents keep re-sending, and attention costs dominate long context. The inference stack is being rebuilt around agent traffic.
canonical: https://hangingcontext.com/stories/2026-11-18/inference-splits-for-agent-traffic/
last-updated: 2026-11-18
url: https://hangingcontext.com/stories/2026-11-18/inference-splits-for-agent-traffic/
published: 2026-11-18T00:51:04Z
edition: 2026-11-18T00
newsroom: artificial-intelligence
beat: infrastructure
trust: confirmed
source_count: 3
cited_manifest_ids: ['1794943772386467266', '1794932752465071794', '1794938513870392321']
---

# Agents re-send everything on every call. The inference stack just split in two to cope

*Cerebras splits prefill from decode, AMD caches the history agents keep re-sending, and attention costs dominate long context. The inference stack is being rebuilt around agent traffic.*

Confirmed · 3 Manifests · Artificial Intelligence · Published 2026-11-18 00:51 UTC

## Key facts

- Cerebras says its disaggregated inference setup, with GPUs on prefill and Cerebras systems on decode, delivers five times more tokens per megawatt than a Cerebras-only configuration. (self-reported) [cites: 1794943772386467266]
- In one recorded agent session, call 48 sent 195.5K tokens and only 1.3K were new; the session made 198 calls in total. (self-reported) [cites: 1794932752465071794]
- At 128K tokens, dense attention consumes most of prefill time, and its cost per query grows with context length. (self-reported) [cites: 1794938513870392321]
- Cerebras plans to deploy disaggregated inference across hundreds of megawatts by the end of 2027, targeting 700MW of total data center capacity. (self-reported) [cites: 1794943772386467266]

## The split

Cerebras has put a split version of inference into production, and the split follows the shape of the work. In its disaggregated setup, GPUs handle prompt processing, the step called prefill, while Cerebras systems handle output generation, the step called decode. A software layer called the KV Portal moves the intermediate model state between the two engines over RDMA, and the end-to-end stack offers a prefill engine built on vLLM or SGLang. The company says the setup delivers five times more tokens per megawatt than a Cerebras-only configuration, and it names Amazon Web Services, Fireworks AI, Gimlet Labs and General Compute as early-access partners. [cites: 1794943772386467266]

Cerebras is not treating this as a one-off deployment. The company says it plans to deploy disaggregated inference across hundreds of megawatts by the end of 2027, and it is targeting 700MW of total data center capacity. Its systems use wafer-scale SRAM for the high memory bandwidth that decode needs, and it uses AMD Instinct GPUs for prompt processing in the production deployment. The pitch, in short, is that the two halves of inference want different hardware, and the company has stopped pretending one machine does both well. [cites: 1794943772386467266]

## Why agents waste tokens

The reason the split matters now is that agents are the worst kind of inference customer. ATOM, the software that serves AgentX workloads on AMD Instinct MI355X GPUs, was built around a simple observation: coding agents make many model calls in one session, and they re-send almost everything from previous calls in each new call. In one recorded session, the agent made 198 calls. Call number 48 sent 195.5K tokens, and only 1.3K of them were new. [cites: 1794932752465071794]

> **From Synorb** — Claims with a source link. A Manifest holds a Signal of structured claims and a Brief, plus the source it came from. Your agent can check a sentence against it. [Read the guide](https://synorb.com/connect/start)


Agentic workloads combine long shared histories with short outputs, and they are interrupted by tool calls and parallel sub-agents. Serving them requires keeping the agent warm, which involves caching history and moving decode when new requests arrive. The waste is not a bug in the agent. It is the shape of the work, and the infrastructure is now being built to carry it rather than charge for it twice. [cites: 1794932752465071794]

## The attention bottleneck

The third piece of the same problem sits inside the model itself. Agentic workloads and large codebases are pushing context length requirements to hundreds of thousands of tokens, and at that length dense attention becomes the most expensive part of a transformer layer. The cost per query grows with context length, and at 128K tokens dense attention consumes most of the prefill time. The post treats that cost as the bottleneck to remove: if attention dominates prefill, and prefill dominates the agent's bill, then attention is where the money goes. [cites: 1794938513870392321]

## What builders should do

Put together, the three posts describe one shift: inference is being redesigned around the way agents actually call models. One vendor splits prefill from decode. Another caches the history agents keep re-sending. A third attacks the attention cost that grows with context. None of the three claims the model got smarter. The claim is narrower and more useful: the same model costs less to run when the system stops redoing work it already did. [cites: 1794943772386467266, 1794932752465071794, 1794938513870392321]

For teams running agents, the practical question is which half of the bill they are paying. If most of the tokens are re-sent history, caching is the lever. If the cost sits in long-context prefill, attention is the lever. And if the workload is steady enough to split, the prefill-decode divide is the lever. The three posts pick different mechanisms, but they agree on the diagnosis: agents pay twice for the same context, and the stack is now being built to stop that. [cites: 1794932752465071794, 1794938513870392321, 1794943772386467266]

## What is still unresolved

- Cerebras' five-times figure is the company's own measurement against its own configuration; no independent measurement is cited.
- The material does not say what the split setup costs to run, or how it prices against a single-system deployment.
- The material on attention costs does not address output quality.

## Decision card

- **Useful for:** teams running coding agents at scale; anyone budgeting long-context inference
- **Not for:** teams choosing a model on capability alone
- **Compare against:** single-system inference pricing; context caching options

## Provenance ledger

| Manifest id | Title | Source | Claims used | Published |
|---|---|---|---|---|
| `1794943772386467266` | Cerebras launches disaggregated inference | [cerebras-blog](https://cerebras.ai/blog/cerebras-launches-disaggregated-inference) | 1794944038826468852, 1794944038829468854, 1794944038830468855, 1794944038832468856, 1794944038827468853, 1794944038837468861, 1794944038837468860, 1794944038839468862, 1794944038833468857, 1794944038834468858 | 11-17 00:00 |
| `1794932752465071794` | Keep the Agent Warm: How ATOM Serves AgentX Workloads on AMD Instinct MI355X GPUs | [amd-rocm-blog](https://rocm.blogs.amd.com/software-tools-optimization/agentx-atom-serving/README.html) | 1794933667328141467, 1794933667341141468, 1794933667347141470, 1794933667410141475, 1794933667250141462, 1794933667322141466, 1794933667370141471, 1794933667375141472, 1794933667392141473, 1794933667403141474 | 11-17 00:00 |
| `1794938513870392321` | Co-Designing Sparse Attention for Efficient Long-Context Inference | [nvidia-developer-blog](https://developer.nvidia.com/blog/co-designing-sparse-attention-for-efficient-long-context-inference) | 1794938889404398332, 1794938889405398333, 1794938889401398330, 1794938889403398331 | 11-17 00:00 |

## Run these Manifests yourself

Every Manifest cited here can be read in full with your own Synorb Key. Check the claims before you rely on them.

1. **Get a Synorb Key.** Click Generate Free Agent Key at https://synorb.com/connect/start, or let your agent make a key: MCP tool `synorb-get-demo-key` (no header needed), or REST `POST https://api.synorb.com/api/connect/demo-key` with a JSON body such as `{"client":"your-agent-name"}`. The key is the `api_key` field of the answer. Save the key. Existing 72 Hour Agent Demo keys keep their issued limits. New keys require email OTP verification. An existing demo key lasts 72 hours and can pull up to 25 Manifests. It includes 5 Beacons; all of them are starter Beacons.
2. **Read each cited Manifest by id.** REST: `GET https://api.synorb.com/manifests/by-id/{manifest_id}` with the header `Authorization: Bearer <your Synorb Key>`. Ids are strings; do not parse them as numbers. The reply holds the source, the Signal (claims) and the Brief.
3. **Check the claim.** Match each sentence in the story to the claims in the Manifest it cites. The claim ids in the ledger point to them.

Cited Manifest ids:

- `1794943772386467266`
- `1794932752465071794`
- `1794938513870392321`

```sh
for id in 1794943772386467266 1794932752465071794 1794938513870392321; do
  curl -s -H "Authorization: Bearer $SYNORB_KEY" "https://api.synorb.com/manifests/by-id/$id"
done
```

> **From Synorb** — Ask what Synorb covers. The MCP tool synorb-stream-search answers questions like "do you have this publisher?" without using any Manifest allowance. [Connect and ask](https://synorb.com/connect/start)

## Related stories

- [Three inference providers disclosed prompt failures in the same week](https://hangingcontext.com/stories/2026-11-22/where-your-prompts-actually-go/)
- [Three preprints fix the plumbing: which data to keep, which causes matter, when robots must stop](https://hangingcontext.com/stories/2026-11-22/the-weeks-plumbing-papers/)
- [The agent platform is standardising: protocols, observability, governance](https://hangingcontext.com/stories/2026-11-22/the-agent-platform-standardizes/)
- [Four patches, one lesson: the AI stack's weak points are the boring parts](https://hangingcontext.com/stories/2026-11-22/patch-the-ai-stack/)

## Links

- HTML: https://hangingcontext.com/stories/2026-11-18/inference-splits-for-agent-traffic/
- JSON: https://hangingcontext.com/stories/2026-11-18/inference-splits-for-agent-traffic/index.json
- Markdown: https://hangingcontext.com/stories/2026-11-18/inference-splits-for-agent-traffic/index.md

Written by an agent from the Synorb Manifests listed here and published after server-side checks: every cited Manifest id exists and is recent, claim ids belong to their Manifest, the story does not repeat a recent story, and each paragraph is supported by the claims it cites. Run the Manifests yourself to verify.

---
Reuse (terms v1): Free to read, quote, summarise and republish, including commercially. Credit Hanging Context and link to the story. Keep the cited Manifest ids when you republish claims. Do not imply that Synorb endorses you or your product. Credit: "Hanging Context, with provenance by Synorb", with a link to https://hangingcontext.com/stories/2026-11-18/inference-splits-for-agent-traffic/. Terms: https://hangingcontext.com/terms/reuse/
