Before you run the agent, the bill is getting predictable

A new yardstick for capacity

Epoch AI Research has introduced the Inference Performance Index, a measure of inference capacity that ranks memory and communication throughput above raw compute speed. The index's first version focuses on agentic workloads and scores hardware in GB300 equivalents, with NVIDIA's GB300 as the reference point. The reason, Epoch AI says, is that inference improvements can be much larger than changes in raw compute speed suggest: the GB300 provides about 25 times the inference capacity of the H100 per chip on the agentic workloads it models, with only 2.5 times the 8-bit floating-point performance. Comparing floating-point performance alone would underestimate the GB300's inference capacity relative to the H100 by a factor of ten. m:…6790

The gap comes from architecture, not just silicon. The GB300 NVL72 connects groups of 72 GPUs through high-bandwidth links, against groups of eight for the B300 HGX, and Epoch AI estimates the NVL72 has about 1.9 times the modeled inference capacity per GPU of the B300. On the index's scale the GB300 NVL72 scores 1, the H200 HGX scores 0.12, the B300 HGX 0.54, AMD's MI355X about 0.55 and the VR200 2.18. The index models language-model inference in two phases — prefill, which processes the input and stores intermediate results in a key-value cache, and decode, which generates the response — and it is built for the workload that now dominates token use: agentic API keys consumed roughly seven times as many tokens as human API keys in late September 2026. Epoch AI notes the index does not account for software maturity, which can change real-world performance. m:…6790

Stop recomputing the past

The cheapest token is the one an agent never recomputes. KV caching stores the attention state of an agent's growing history so each call does not recompute it, FriendliAI explains, and in a 1,000-task example reusing cached history lowered input token costs by 80%. The company describes four mechanisms: prefix caching, which reuses retained state when a later request begins with an identical token sequence; hierarchical caching, which offloads KV state to host DRAM and disk to extend capacity beyond GPU memory; cache-aware routing, which sends each request to a worker that already holds the matching history; and sparse attention, which selectively reads from the main KV cache to cut memory traffic. m:…8665

Two preprints push the same idea further. LowRAM compresses KV caches by predicting values from keys within each attention head and applying low-rank key compression across layers, reaching up to five times higher peak decode throughput at four times compression and, combined with quantization and eviction, 26.9 times compression at 94.5% of baseline performance on long contexts. LaProx, a second preprint, reframes cache eviction as an output-aware, layer-wise problem and reports maintaining model performance with only 5% of the KV cache, with up to twice the accuracy-loss reduction under extreme compression compared with existing baselines. The numbers are the papers' own, and both need independent replication — but they agree on the direction: the cache, not the model call, is where agent inference gets cheap. m:…5407 m:…5090

Predict the bill before routing

The last step is knowing the cost before choosing the model. In a preprint, researchers propose a small, frozen scout language model that reads prefill activations to predict a larger reasoning model's chance of success and its output token count. With perfect per-query cost knowledge, the paper reports savings of 29 to 61% of spending on eight of nine datasets; with the scout's predictions, savings of 9 to 33% at matched accuracy against median-length pricing. Savings plateau at a scout of about 2 billion parameters, the authors report, which gives infrastructure teams a simple rule: a scout bigger than that buys little. Output length tracks problem difficulty closely in math and code datasets, but not in knowledge or mixed-format ones, so the scout needs different probes for different kinds of task. m:…1204

What this changes

Put together, the three threads answer different questions a team asks before running agents at scale: how much capacity does this hardware really have for agent work, how much of the bill is repeated computation, and how long will this particular task run. The index answers the first, the cache work the second and the scout the third. None of them cuts a price on its own. What they cut is uncertainty — and for a team deciding how many agents to run, or which hardware to buy for them, uncertainty is the line item that has been hardest to budget. m:…6790 m:…8665 m:…1204