Agents don't need bigger models. They need somewhere to put the conversation.

The cache is the bottleneck

A chatbot asks one question and waits. An agent asks, acts, reads the result, asks again, and keeps the whole conversation in memory while it does. That difference is now the central problem in inference serving, and a preprint posted this week names it precisely: KV cache thrashing is a primary cause of performance degradation in concurrent, multi-turn agent workflows on memory-constrained devices. The paper's system, MARS, adds an admission controller for root workflows and an agent-centric scheduler for turn completion, and reports 1.59x to 7.56x mean end-to-end latency speedups over the fastest baseline. Plugged into OpenCode, Codex and OpenHands, it improved mean workflow completion time by 1.17x to 1.87x. The cache, not the model, is what slows the agent down. m:…8153

A second preprint, Evolving Infra, attacks the same bottleneck from the architecture side. It decouples compute allocation from KV cache management so nodes can switch between prefill and decode roles as the workload changes, and adds an agent-aware distributed KV manager that guides cache eviction and transfer using signals from agent context management. The reported gains are large: mean end-to-end request latency down by up to 4.1x and P95 time-to-first-token down by up to 14x compared with NVIDIA's Dynamo. The premise of both papers is the same — serving systems designed for single-turn chat are a poor fit for agents that hold long, branching conversations. m:…8147

Squeeze it, move it, share it

If the cache is the constraint, one answer is to make it smaller. OSCAR, a third preprint, uses offline spectral covariance estimation to derive fixed per-layer rotations and clipping thresholds for 2-bit KV-cache quantisation. It reports 8x KV-cache memory compression and up to 7.8x higher multi-request throughput at large batch sizes compared with BF16, while narrowing the accuracy gap to 3.78 and 1.21 points on Qwen3-4B and 8B models respectively. The implementation uses fused Triton kernels and stays compatible with SGLang and vLLM, which is the part that matters for adoption: a compression trick nobody can deploy is a paper, not a product. m:…8163

Another answer is to move the work to cheaper hardware. HAWK splits the two phases of inference across device types, mapping prefill to GPUs and decode to NPUs, with an RDMA-based KV cache transfer mechanism that resolves the tensor layout differences between them. The system achieves performance within 7.88 percent of a GPU-only baseline under the same service-level constraints, while delivering 1.26x higher power efficiency and 1.72x higher cost efficiency. Decode is memory-bound rather than compute-bound, so running it on expensive GPUs is paying for flops the workload does not use. m:…8150

The third answer is coordination. Tensormesh says its new LMCache coordinator acts as a control plane that unifies KV cache servers across a fleet into a single manageable pool, with a fleet-wide view of cache events and pluggable controllers for eviction, prefetching and placement policies. Crucially, the coordinator is kept out of the inference path — servers check with it only when necessary — so inference continues even if the coordinator goes offline. The cache stops being a per-server accident and becomes a shared resource with policy. m:…0338

The long tail, and the radical fix

A fourth preprint looks at batch work, where agents run hundreds of tasks at once for synthetic data or evaluation. Its Chain-Advancing Action Scheduling prioritises long agent chains to shorten the batch tail, and coordinated optimisation of GPU memory between KV cache and recurrent states reduces batch makespan. Decoupling sample admission from environment execution improved resource utilisation. The lesson is the same as in the serving papers: agent workloads alternate between thinking and acting, and a system that treats those as one job wastes the GPU on one of them. m:…8157

The most radical paper of the week removes the memory from the GPU altogether. Optical Broadcast Memory replaces local HBM weight storage with a centralised, fiber-based broadcast system that sends weights to distributed inference nodes. At a scale of 80,000 accelerators, the authors report, it removes the need for local HBM weight storage while achieving competitive or superior throughput and energy-per-token metrics compared with traditional HBM-based systems, and it supports mixture-of-experts models by removing the fixed-length constraints of older optical ring designs. It is the furthest from production of the week's ideas, and the clearest sign of where the pressure is: the serving stack is being redesigned around agents, and the redesign starts with the memory. m:…8166