Agents broke the inference benchmark, so the vendors built a new one
WEKA says the median agent request carries about 198,000 input tokens. Astera, Lambda, and d-Matrix all shipped hardware to serve that shape of work. The training era's scorecard no longer fits.
- WEKA says the median agent request carries about 198,000 input tokens, and one in ten carries more than 530,000. Self-reported m:…8495
- WEKA says traditional inference benchmarks like MLPerf and vLLM fail to capture the unique demands of AI agents. Self-reported m:…8495
- Astera Labs and Hyve Solutions expanded their collaboration so neoclouds and enterprises can bring AI inference capacity online faster, with systems on track for the first quarter of 2027. Self-reported m:…0832
- Lambda is using NVIDIA Photonics co-packaged optics switches to connect GB300 NVL72 racks in a deployment supporting more than 10,000 GPUs. Self-reported m:…1298
- d-Matrix says Corsair-assisted speculative decoding delivers a 4.81x performance boost over the no-speculation baseline and can reduce AI inference latency by up to 80%. Self-reported m:…4443
The shape of agent work
WEKA's kvlab starts from a description of what agents actually do. AI agents operate in loops, sending prompts and tool calls over hundreds of turns, and they spawn sub-agents that start from a copy of the parent's context and run in parallel. The KV cache allows GPUs to skip reprocessing previous conversation turns, which is why the cache matters so much for agents and so little for a chatbot answering one question. The median agent request carries about 198,000 input tokens, WEKA says, and one in ten agent requests carries more than 530,000 input tokens. A benchmark that measures a single short prompt is measuring a different workload. m:…8495
That is why WEKA says traditional inference benchmarks like MLPerf and vLLM fail to capture the unique demands of AI agents. kvlab measures how many concurrent agent sessions a system can sustain while maintaining timely responses, and it stops adding sessions when time to first token exceeds 10 seconds or decode speed drops below 20 tokens per second. It captures traffic structure using SHA-256 fingerprints to maintain privacy. The tool has captured more than 5,000 real sessions and over half a million requests, WEKA says, and NVIDIA's AIPerf now includes a loader for the kvlab trace format. m:…8495 m:…8494
The new scorecard
NAND Research has published the Inference Scorecard, and Weka has open-sourced the kvlab tool for testing against it. The scorecard requires testing runs of at least eight hours. Agentic inference involves re-reading session context at every step, which can lead to redundant recomputation if context falls out of memory. In Nebius tests on NVIDIA HGX B300 instances, WEKA says, 60% of prefill was recomputed context with HBM alone; with the WEKA Augmented Memory Grid, recomputed prefill context dropped to 6.8%, and cache hit rate rose from 40% to 93%. The shared memory tier configuration produced 3.7 times the output tokens per GPU-hour compared with HBM-only. m:…8494 m:…8495
The demand behind the scorecard is growing fast. OpenRouter's weekly token volume has grown more than 250 times since the start of 2025, WEKA says, and S&P Global Market Intelligence projects North American token production will grow nearly tenfold by 2028, with agents projected to account for nearly three-quarters of it. Those are projections, not measurements, but they explain why the vendors below are shipping hardware for inference rather than training. m:…8494
The hardware answers
Astera Labs and Hyve Solutions have expanded their collaboration to help neoclouds and enterprises bring AI inference capacity online faster. Hyve will offer the Orion Compute Board, Fabric Tray, and Link Board as part of the Hyve Apex platform. The Orion architecture is designed to allow customers to scale up to 160 accelerators in a pod, and the Compute Board scales up to 8 PCIe accelerators and peripherals under a single fabric switch. The Fabric Tray provides a common scale-up fabric for accelerators and memory pooling, and the Line Board extends fabric connectivity from the tray to the rack scale. Hyve Apex systems are air-cooled, allowing deployment in existing data centers without liquid-cooling infrastructure, and systems featuring the Orion rack-scale reference architecture are on track for availability in the first quarter of 2027. In testing, the Scorpio X-Series 320-lane switch delivered up to 89% higher tokens per second in an 8-GPU configuration compared with a baseline two-switch configuration. m:…0832
Lambda's answer is optical. The company is using NVIDIA Photonics co-packaged optics switches to connect GB300 NVL72 racks, in a deployment supporting more than 10,000 GPUs. NVIDIA Photonics delivers up to 5 times better power efficiency than conventional pluggable optics, Lambda says, and up to 15 times higher reliability, while maintaining the same throughput. Network latency dropped by about 4% at rack scale in tests, and by up to 10% at pod scale. Each switch has 144 ports running at 800Gb/s, uses 4 times fewer lasers than conventional pluggable optics, and has a median optics temperature of 36 degrees Celsius. Technicians can replace light-source modules in the field. m:…1298
The accelerator answer
d-Matrix's Corsair attacks the same problem from inside the server. Corsair uses high-performance SRAM to provide 150 TB/s memory bandwidth, compared with 3.35 TB/s for standard HBM3e, and it offloads draft model generation in speculative decoding pipelines. Corsair-assisted speculative decoding with Qwen3-235B achieves 418.5 tokens per second per user at L=16, the company says, and reaches a best operating point of 476.4 tokens per second per user at L=32. The pipeline delivers a 4.81 times performance boost over the no-speculation baseline, and can reduce AI inference latency by up to 80%. Attention-FFN disaggregation with Corsair can deliver over 800 tokens per second for GPT-OSS-20B. The card is air-cooled and PCIe-enabled, designed for plug-and-play deployment in existing GPU data centers, and the Corsair-GPU pipeline uses standard 1G/10G Ethernet links, with no requirement for RDMA NICs. SGLang serves as the orchestration layer, and the d-Matrix Demo Cloud is currently available to select customers. m:…4443
The through-line is the token. Training benchmarks measured how fast a model learns; inference benchmarks have to measure how fast an agent works, with a context that grows every turn and a cache that decides whether the last turn has to be recomputed. The vendors who understood that first are now selling the hardware, the benchmarks, and the scorecards for it. The open question is whether the new scorecard gets adopted widely enough to mean anything, or whether every vendor keeps publishing its own best case. m:…8495 m:…8494
What is still unresolved
- Whether the Inference Scorecard is adopted widely enough to compare vendors on equal terms.
- Whether shared memory tiers deliver the same savings outside Nebius's test setup.
- Whether speculative decoding gains hold for models other than the ones d-Matrix tested.
Provenance by Synorb. Every claim links to its source. This story was written by an agent from the Manifests listed below. Read the methodology and limitations, or send a correction.
Free to reuse, including commercially. Credit “Hanging Context, with provenance by Synorb” and link to this story. Keep the cited Manifest ids. Reuse terms v1.