The KV cache is becoming a memory hierarchy

Shrink the cache

In a preprint, AbridgeKV attacks the memory-bound nature of long chain-of-thought decoding by training models to compress generated text into learned pause tokens. Once the reasoning is compressed, the original cache entries can be evicted without accuracy loss, effectively decoupling test-time compute from memory growth. The reported numbers are a 3.5x reduction in cache size per sequence and a 3.2x speedup on H100 hardware versus full-KV decoding, with the method outperforming existing eviction-based cache management at 32K-token generations. m:…3063

RoLaQ takes the complementary route: instead of compressing what is stored, store less per token. In a preprint, its joint rotation and range calibration method targets 2-bit KV cache quantization for Multi-head Latent Attention, the architecture behind DeepSeek-style models. The authors report 3.96x the BF16 KV capacity under the same memory budget, 2.51x decode throughput with 256K input tokens on DeepSeek-V2-Lite, and wins over existing 2-bit quantization methods across six benchmarks. Long context on memory-constrained hardware stops looking like a luxury. m:…7088

Read less of it

In a preprint with an unusually direct title, Language Models Can Control Their Own Attention, the authors let the model declare how it wants to attend. Declarative Attention partitions generation into context modes so the model can bypass scanning the entire KV cache, cutting total attended tokens per response by 45.8% for Gemma-4-31B and 19.6% for Qwen-3.6-27B. The price is an accuracy drop of 1.40 and 2.10 percentage points respectively. m:…8006

The detail that matters for roadmaps is that the accuracy penalty shrinks as model scale increases. This suggests, and this is reasoning rather than a reported result, that attention selectivity becomes a free or nearly free lunch at frontier scale. If that holds, the default architecture of 2027 reads a fraction of its own context by design, and the KV cache is consulted more like an index than a tape. m:…8006

Move it like infrastructure

The third front treats the KV cache as a distributed systems problem. In a preprint, EdgeFlow addresses stateful KV cache migration in geo-distributed edge environments, where bandwidth is constrained and servers are unreliable. By dynamically balancing direct cache transmission against activation-based recomputation, with a 2-approximation algorithm optimizing multi-node migration, the authors report cutting migration time by up to 45%. Stateful inference that survives node failure without dropping the conversation is becoming an operations discipline. m:…2916

And a companion preprint issues the warning label. Naive remote KV tiers can be slower than recomputation: synchronous loading turns hit-heavy streams into unstable queues, pushing p99 latency from 3.96s to 17.3s for 16k-token prompts. Remote hits only pay off above a bandwidth threshold the authors derive as 2.53 GB/s for their configuration; their sharded, asynchronous architecture with a rank-partitioned tier lifts GET bandwidth 6.3x. The takeaway for operators is blunt: disaggregated inference needs its own memory hierarchy design, not just a network mount on the cache. m:…8642

Step back and the pattern is clear, and this is reasoning rather than a reported result. Five preprints, three directions, one conclusion: the KV cache is becoming the memory hierarchy of the inference era, with its own compression layer, its own access patterns, and its own network tier. Teams that budget for GPUs but not for cache bandwidth and migration are planning for last year's bottleneck. The winners will be the serving stacks that treat memory, not FLOPs, as the scarce resource. m:…3063 m:…7088 m:…2916