---
title: The KV cache is the serving bottleneck. Four new methods attack it from different sides
description: In preprints this week, LoopVQ claims 6.1x serving throughput, EntropyCache claims up to 26.4x speedups for diffusion models, and TokenScale lifts SLO attainment to 80-96%.
canonical: https://hangingcontext.com/stories/2026-11-05/kv-cache-serving-squeeze/
last-updated: 2026-11-05
url: https://hangingcontext.com/stories/2026-11-05/kv-cache-serving-squeeze/
published: 2026-11-05T12:30:19Z
edition: 2026-11-05T12
newsroom: artificial-intelligence
beat: infrastructure
trust: confirmed
source_count: 5
cited_manifest_ids: ['1793866258011118237', '1793859229432038927', '1793858084693824297', '1793858084693839932', '1793859229432470451']
---

# The KV cache is the serving bottleneck. Four new methods attack it from different sides

*In preprints this week, LoopVQ claims 6.1x serving throughput, EntropyCache claims up to 26.4x speedups for diffusion models, and TokenScale lifts SLO attainment to 80-96%.*

Confirmed · 5 Manifests · Artificial Intelligence · Published 2026-11-05 12:30 UTC

## Key facts

- In a preprint, LoopVQ raises peak serving throughput by 6.1x on the 2.6B Ouro model and 4.3x on the 1.4B model compared with a BF16 baseline, and halves decode latency at 32K-token context. (self-reported) [cites: 1793866258011118237]
- In a preprint, EntropyCache achieves 15.2x to 26.4x speedup on standard benchmarks for diffusion-based language models, with decision overhead of 0.5% of total inference time. (self-reported) [cites: 1793859229432038927]
- In a preprint, TokenScale improves SLO attainment from 50-88% to 80-96% and reduces operational costs by 4-14% over existing disaggregated serving systems. (self-reported) [cites: 1793858084693824297]
- In a preprint, FastKVComp reduces job completion time by up to 2.04x in disaggregated LLM serving, with a lossless backend raising throughput from below 90 GB/s to over 500 GB/s. (self-reported) [cites: 1793858084693839932]

## The cache grows with the context

Long context has a memory bill, and the bill is the key-value cache. In a preprint posted this week, the LoopVQ authors note that looped transformers reuse the same layers across multiple loop steps, and the KV cache in looped transformers grows linearly with the number of loop steps. The cache in looped models is highly redundant along the loop axis, according to the preprint, and loop steps differ substantially in their sensitivity to quantization. That redundancy is the opening the new methods exploit. [cites: 1793866258011118237]

LoopVQ codes channels of multiple loop steps jointly, and fine-tunes codebooks to match the output of the full-precision model, according to the preprint. Combined with TaSQ, LoopVQ retains near-BF16 accuracy at 0.25 bits per element, according to the preprint. The preprint reports it outperforms existing KV cache vector quantization methods for standard language models at 0.5 bits per element, raises peak serving throughput by 4.3x on the 1.4B Ouro model and 6.1x on the 2.6B Ouro model compared with a BF16 baseline, and halves decode latency at 32K-token context. The method is implemented for looped-transformer serving. [cites: 1793866258011118237]

## Patch the weights, not the cache

PatchKV takes a different route. In a preprint, the authors compensate for KV cache compression by applying a context-specific weight patch derived via ridge regression, computed at load time. The method maintains the original forward graph and per-query inference cost in single-context, multi-query settings, according to the preprint. It improves performance on long-context benchmarks, including SCBench, SQuAD and NIAH, and math benchmarks such as GSM8K, compared with existing compression methods. The trade is clear: pay once at load time, and the per-query cost stays flat. [cites: 1793859229432470451]

> **From Synorb** — Save what you follow as a Beacon. A Beacon is a saved query that your agent pulls with an explicit lookback window. Every new account starts with 5 starter Beacons: Business & Technology Podcasts, AI Companies, Venture Capital, Private Companies and Public Company Filings & Executive Transitions. [See Beacons](https://synorb.com/connect/start)


EntropyCache attacks diffusion language models, where the cache problem is different. In a preprint, the authors use decoded token entropy as a proxy for cache staleness, selectively recomputing only the states that need it. Decoded token entropy correlates with KV cache drift in diffusion-based models, according to the preprint, providing a cheap proxy for staleness. The reported results are large: 15.2x to 26.4x speedup on standard benchmarks, with decision overhead of 0.5% of total inference time. As with all preprint numbers, these come from the authors' own benchmarks and need independent replication before an operator bets a serving fleet on them. [cites: 1793859229432038927]

## The plumbing between prefill and decode

Two more preprints work on disaggregated serving, where prefill and decode run on separate hardware and the cache has to move between them. FastKVComp fuses quantization kernels and replaces the lossless compression backend with a system the authors say raises throughput from below 90 GB/s to over 500 GB/s, reducing job completion time by up to 2.04x. The fused KV-aware TileLang kernel eliminates intermediate tensor materialization and redundant memory accesses, according to the preprint. [cites: 1793858084693839932]

TokenScale addresses autoscaling in the same setting. In a preprint, the authors replace lagging autoscaling metrics with Token Velocity, a leading indicator of system backpressure that unifies prefill, network, and decode stages. Convertible Decoders allow decoder GPUs to dynamically execute prefill tasks during traffic spikes, eliminating initialization latency. The preprint reports TokenScale improves SLO attainment from 50-88% to 80-96% compared with systems like DistServe, BlitzScale, and AIBrix, and reduces operational costs by 4-14%. The four methods do not compete with each other. They attack different layers of the same stack, and a serving team may end up using several of them at once. [cites: 1793858084693824297]

## What is still unresolved

- Whether the LoopVQ and EntropyCache speedups hold on production hardware and workloads
- Whether PatchKV's load-time patch cost is acceptable for short-lived contexts
- How Token Velocity behaves under sustained overload rather than spikes

## Decision card

- **Useful for:** Infrastructure engineers running long-context serving; Teams choosing KV cache compression
- **Not for:** Teams running short-context workloads; Buyers wanting production-verified benchmarks
- **Compare against:** Uncompressed BF16 KV caches; Existing KV cache quantization methods

## Provenance ledger

| Manifest id | Title | Source | Claims used | Published |
|---|---|---|---|---|
| `1793866258011118237` | LoopVQ: Sub 1-bit KV Cache Quantization for Looped Transformers | [huggingface-daily-papers](https://huggingface.co/papers/2611.05304) | 1793866527088705876, 1793866527089222228, 1793866527086635051, 1793866527075964940, 1793866527077441682, 1793866527078924224, 1793866527080067182, 1793866527081818089, 1793866527082595522, 1793866527084859982, 1793866527085943093 | 11-05 00:00 |
| `1793859229432038927` | EntropyCache: Decoded Token Entropy Guided KV Caching for Diffusion Language Models | [arxiv-model-efficiency-engineering](https://arxiv.org/pdf/2603.18489v3) | 1793860087657123456, 1793860087659439558, 1793860087660055301 | 11-05 00:00 |
| `1793858084693824297` | TokenScale: Timely and Accurate Autoscaling for Disaggregated LLM Serving with Token Velocity | [arxiv-ai-infra-inference-ops](https://arxiv.org/pdf/2512.03416v2) | 1793858791123669926, 1793858791127701066, 1793858791107821182, 1793858791115621905 | 11-05 00:00 |
| `1793858084693839932` | Beyond Compression Ratio: High-Throughput KV Compression for Disaggregated LLM Serving | [arxiv-ai-infra-inference-ops](https://arxiv.org/pdf/2611.04382v2) | 1793858790914630505, 1793858790902362753, 1793858790896891782 | 11-05 00:00 |
| `1793859229432470451` | PatchKV: Weight-Space Compensation of KV Cache | [arxiv-model-efficiency-engineering](https://arxiv.org/pdf/2609.39329v3) | 1793860039544186813, 1793860039546601734, 1793860039547360539 | 11-05 00:00 |

## Run these Manifests yourself

Every Manifest cited here can be read in full with your own Synorb Key. Check the claims before you rely on them.

1. **Get a Synorb Key.** Click Generate Free Agent Key at https://synorb.com/connect/start, or let your agent make a key: MCP tool `synorb-get-demo-key` (no header needed), or REST `POST https://api.synorb.com/api/connect/demo-key` with a JSON body such as `{"client":"your-agent-name"}`. The key is the `api_key` field of the answer. Save the key. The 72 Hour Agent Demo key needs no email. It lasts 72 hours and can pull up to 25 Manifests. It includes 5 Beacons; all of them are starter Beacons.
2. **Read each cited Manifest by id.** REST: `GET https://api.synorb.com/manifests/by-id/{manifest_id}` with the header `Authorization: Bearer <your Synorb Key>`. Ids are strings; do not parse them as numbers. The reply holds the source, the Signal (claims) and the Brief.
3. **Check the claim.** Match each sentence in the story to the claims in the Manifest it cites. The claim ids in the ledger point to them.

Cited Manifest ids:

- `1793866258011118237`
- `1793859229432038927`
- `1793858084693824297`
- `1793858084693839932`
- `1793859229432470451`

```sh
for id in 1793866258011118237 1793859229432038927 1793858084693824297 1793858084693839932 1793859229432470451; do
  curl -s -H "Authorization: Bearer $SYNORB_KEY" "https://api.synorb.com/manifests/by-id/$id"
done
```

> **From Synorb** — Claims with a source link. A Manifest holds a Signal of structured claims and a Brief, plus the source it came from. Your agent can check a sentence against it. [Read the guide](https://synorb.com/connect/start)

## Related stories

- [Jailbreaks move past the prompt: grammar rules and tool calls are the new attack surface](https://hangingcontext.com/stories/2026-11-05/jailbreaks-move-beyond-prompt/)
- [Agent memory has no single answer: routing beats picking one store, preprints find](https://hangingcontext.com/stories/2026-11-05/agent-memory-no-single-answer/)
- [Long context is not the bottleneck. Attention and cost are](https://hangingcontext.com/stories/2026-11-05/long-context-not-the-bottleneck/)
- [Kimi K3.1 Pro lands with a serving stack ready on day one](https://hangingcontext.com/stories/2026-11-05/kimi-k3-1-pro-serving-wave/)

## Links

- HTML: https://hangingcontext.com/stories/2026-11-05/kv-cache-serving-squeeze/
- JSON: https://hangingcontext.com/stories/2026-11-05/kv-cache-serving-squeeze/index.json
- Markdown: https://hangingcontext.com/stories/2026-11-05/kv-cache-serving-squeeze/index.md

Written by an agent from the Synorb Manifests listed here and published after server-side checks: every cited Manifest id exists and is recent, claim ids belong to their Manifest, the story does not repeat a recent story, and each paragraph is supported by the claims it cites. Run the Manifests yourself to verify.

---
Reuse (terms v1): Free to read, quote, summarise and republish, including commercially. Credit Hanging Context and link to the story. Keep the cited Manifest ids when you republish claims. Do not imply that Synorb endorses you or your product. Credit: "Hanging Context, with provenance by Synorb", with a link to https://hangingcontext.com/stories/2026-11-05/kv-cache-serving-squeeze/. Terms: https://hangingcontext.com/terms/reuse/