The KV cache is the serving bottleneck. Four new methods attack it from different sides

The cache grows with the context

Long context has a memory bill, and the bill is the key-value cache. In a preprint posted this week, the LoopVQ authors note that looped transformers reuse the same layers across multiple loop steps, and the KV cache in looped transformers grows linearly with the number of loop steps. The cache in looped models is highly redundant along the loop axis, according to the preprint, and loop steps differ substantially in their sensitivity to quantization. That redundancy is the opening the new methods exploit. m:…8237

LoopVQ codes channels of multiple loop steps jointly, and fine-tunes codebooks to match the output of the full-precision model, according to the preprint. Combined with TaSQ, LoopVQ retains near-BF16 accuracy at 0.25 bits per element, according to the preprint. The preprint reports it outperforms existing KV cache vector quantization methods for standard language models at 0.5 bits per element, raises peak serving throughput by 4.3x on the 1.4B Ouro model and 6.1x on the 2.6B Ouro model compared with a BF16 baseline, and halves decode latency at 32K-token context. The method is implemented for looped-transformer serving. m:…8237

Patch the weights, not the cache

PatchKV takes a different route. In a preprint, the authors compensate for KV cache compression by applying a context-specific weight patch derived via ridge regression, computed at load time. The method maintains the original forward graph and per-query inference cost in single-context, multi-query settings, according to the preprint. It improves performance on long-context benchmarks, including SCBench, SQuAD and NIAH, and math benchmarks such as GSM8K, compared with existing compression methods. The trade is clear: pay once at load time, and the per-query cost stays flat. m:…0451

EntropyCache attacks diffusion language models, where the cache problem is different. In a preprint, the authors use decoded token entropy as a proxy for cache staleness, selectively recomputing only the states that need it. Decoded token entropy correlates with KV cache drift in diffusion-based models, according to the preprint, providing a cheap proxy for staleness. The reported results are large: 15.2x to 26.4x speedup on standard benchmarks, with decision overhead of 0.5% of total inference time. As with all preprint numbers, these come from the authors' own benchmarks and need independent replication before an operator bets a serving fleet on them. m:…8927

The plumbing between prefill and decode

Two more preprints work on disaggregated serving, where prefill and decode run on separate hardware and the cache has to move between them. FastKVComp fuses quantization kernels and replaces the lossless compression backend with a system the authors say raises throughput from below 90 GB/s to over 500 GB/s, reducing job completion time by up to 2.04x. The fused KV-aware TileLang kernel eliminates intermediate tensor materialization and redundant memory accesses, according to the preprint. m:…9932

TokenScale addresses autoscaling in the same setting. In a preprint, the authors replace lagging autoscaling metrics with Token Velocity, a leading indicator of system backpressure that unifies prefill, network, and decode stages. Convertible Decoders allow decoder GPUs to dynamically execute prefill tasks during traffic spikes, eliminating initialization latency. The preprint reports TokenScale improves SLO attainment from 50-88% to 80-96% compared with systems like DistServe, BlitzScale, and AIBrix, and reduces operational costs by 4-14%. The four methods do not compete with each other. They attack different layers of the same stack, and a serving team may end up using several of them at once. m:…4297