Inference serving's next leap is not a faster chip — it is using less memory per token

Stop storing the weights everywhere

In a preprint this week, researchers propose Optical Broadcast Memory, an architecture that eliminates the need for local HBM weight storage by using a centralized memory server to broadcast weights to distributed inference nodes over fiber. At a scale of 80,000 accelerators, the design achieves competitive or superior throughput and energy-per-token metrics compared with traditional HBM-based systems, according to the authors,. It also supports efficient serving of mixture-of-experts models by removing the fixed-length constraints of traditional optical ring architectures. The bet is that moving weights once, to everyone, beats storing a copy next to every accelerator. m:…8166

A second preprint attacks the same memory wall from inside the GPU. SlipStream addresses high-bandwidth flash for LLM inference with a software-hardware co-designed prefetcher, replacing cache-line prefetching with a demand-paced producer-consumer pipeline. It achieves 92.4 percent of ideal HBF performance using a 40 MB staging buffer, significantly outperforming H3, which reaches 43.1 percent with a 320 MB buffer. It supports mixture-of-experts models too, maintaining 91.2 percent of ideal performance with a 40 MB buffer. The pattern is the same as the broadcast paper: the scarce resource is memory bandwidth and capacity, and the win comes from using far less of it. m:…8167

Stop wasting the GPU between tokens

Two more preprints attack idle time rather than memory. SubGPU Pipelining partitions operator graphs into segments bound to disjoint pools of streaming multiprocessors inside a single GPU, so bursty decode kernels stop leaving hardware idle. It provides up to 2.2 times higher per-user throughput than temporal execution at matched per-GPU throughput, and up to 1.9 times higher per-GPU throughput at matched per-user throughput. On B200 hardware it improves DRAM read-plus-write throughput from 71.8 percent to 83.6 percent of peak and reduces 36-layer decoder iteration time by 10.33 percent. The GPU was already fast; it was just waiting. m:…8161

DREX takes on batched early-exit inference, where tokens that could leave the model early are forced to wait for the full pass. By implementing cross-iteration batching, DREX lets tokens wait for deep-layer passes without stalling the entire pipeline, achieving 2.22 to 2.24 times decode speedup over non-early-exit baselines on Llama-2-70B. Output quality stays within 1.4 ROUGE-2 points of non-early-exit models on CNN/DM, and the speedups are 9 to 63 percent larger than EAGLE across evaluated model sizes. Early exit only pays if the batching is right, and this preprint claims to have fixed the batching. m:…8159

Agents break the old serving model

The fifth preprint, Evolving Infra, argues that agentic workloads break disaggregated serving altogether, because agents are dynamic in ways a single chat completion is not. It decouples compute allocation from KV cache management, letting nodes switch efficiently between prefill and decode roles, and adds an agent-aware distributed KV manager that optimizes cache eviction and transfer based on agent context. The reported result is a reduction in mean end-to-end request latency by up to 4.1 times and P95 time-to-first-token by up to 14 times compared with NVIDIA's Dynamo. That is the largest claim in the set, and the most dependent on the workload matching the authors' agent mix. m:…8147

The industry context matches. At KubeCon NA 2026, which added a dedicated AI Inference and Agentic track, the lesson reported was that frontier-scale teams split LLM requests into prefill and decode tasks on separate GPU pools, that moving and sharing the KV cache is a primary technical challenge for large model deployments, and that stateful agents exhibit idle-then-burst traffic patterns that differ from traditional web apps. The preprints are, in effect, five different answers to the KubeCon problem: broadcast the weights, prefetch smarter, split the GPU, exit early, and serve agents as agents. All five are preprints, none peer-reviewed, and their speedups are the authors' own measurements. But they agree on where the money goes next: not a faster chip, but less memory and less idle time per token. m:…0210 m:…8166 m:…8147