The serving stack's memory wars: unified KV addressing hits 2.51x, GPU-NPU disaggregation cuts cost 1.72x

One address space for the KV cache

The KV cache is the tax every long-context serving system pays: the longer the context, the more memory each request pins down, and the faster the GPUs fill up. In a preprint, a preprint proposes UlaCache, which gives the KV cache a unified logical address space. The inference engine manages cache blocks across GPU and CPU pools without caring where they physically live, and a parallel bucketed LRU policy, PB-LRU, manages the shared GPU pool by updating recency metadata and selecting eviction victims in parallel. The reported peak is 2.51x decoding throughput on B300 hardware in tensor-parallel mode compared with state-of-the-art baselines. m:…6698

The implication, if that number holds up, is that memory management overhead rather than raw compute is the binding constraint on decode throughput, and that an addressing abstraction can buy back headroom without new hardware. That is the kind of claim worth testing on your own cluster before believing, but it is also the cheapest kind of claim to test: it is software. m:…6698

Decode doesn't need a GPU

HAWK starts from a different observation: GPUs sit underutilized during the decode phase, so why not hand decode to cheaper NPUs? The catch has always been moving the KV cache between devices with different tensor layouts. HAWK's answer is tensor-layout-aware RDMA buffers that let the NPU consume transferred KV cache tensors directly, without an extra data-rearrangement pass. In the authors' evaluation the heterogeneous setup reached 1.72x the cost efficiency and 1.26x the power efficiency of a GPU-only system, while holding inference performance within 7.88% of the GPU-only baseline under identical latency constraints. m:…6718

That 7.88% figure does the important work in the paper. It converts the disaggregation pitch from a latency gamble into a cost question: if serving quality stays within single-digit percent of the baseline while cost efficiency nearly doubles, the GPU-only default gets harder to defend, at least for workloads whose latency budgets tolerate the transfer. As with all five of these papers, the figures are the authors' own measurements. m:…6718

Learn what to keep

KVLearn works the same disaggregated setup, with prefill and decode on separate nodes, but asks a narrower question: once the KV cache is scattered across the cluster, what should you keep? The framework learns the answer. A Prefix Reuse Predictor estimates how likely a cached prefix is to be reused, from structural and temporal features of the prefix, without touching model weights; a cost-aware retention score and an adaptive threshold controller then adjust admission online as memory pressure and hit rates change. The reported results are up to 56% lower time-to-first-token than no caching, 33% lower than Mooncake-style baselines, and 53% less inter-node KV transfer volume than LRU pooling. m:…5234

Profile cheaper, run on CPU

Two more papers work the edges of the same cost problem. Dooly attacks profiling, the simulation runs teams do before choosing a serving configuration. It exploits structural redundancy across configurations: one inference pass plus taint propagation to label input dimensions lets it skip re-profiling operations already in its latency database. The authors report 56.4% fewer profiling GPU-hours than existing approaches, with simulation error within 5% MAPE on time-to-first-token and 8% on time-per-output-token. m:…4125

SubTernary goes the other direction and makes the CPU a viable inference target. Speculative decoding on CPUs is bottlenecked by context-swapping latency and memory allocation overhead, and SubTernary answers with unified graph caching and transactional KV-cache isolation. The reported speedups are 1.35x over llama.cpp's speculative decoding and 3.99x over standard autoregressive decoding. The ablation is the interesting part: systems runtime optimizations drove 35.0% of the throughput gains, while a 256-bit AVX2 ternary SIMD kernel added only 6.0%. The plumbing beat the kernel. m:…3118