{ "schema": "hangingcontext.desk/1", "type": "story", "id": "74", "slug": "kv-cache-diet", "edition": "2026-10-15T12", "url": "https://hangingcontext.com/stories/2026-10-15/kv-cache-diet/", "newsroom": "artificial-intelligence", "beat": "infrastructure", "newsroom_title": "Artificial Intelligence", "audience": "ai_developer", "angle": "The KV cache goes on a diet: extreme quantization comes for inference memory", "headline": "The KV cache goes on a diet: INT2 quantization nears lossless", "dek": "Four new preprints attack LLM inference memory from four sides: crush the KV cache to INT2, protect the attention sinks, allocate bits by saliency down to 1.5 bits, or evict smarter. The numbers are striking; all are unreplicated preprint claims.", "published_at": "2026-10-15T12:45:31Z", "labels": { "trust": "confirmed", "trust_text": "Confirmed · 4 sources", "newsroom": "Artificial Intelligence" }, "source_count": 4, "publishers": [ "arxiv-frontier-methods-select", "arxiv-model-efficiency-engineering", "huggingface-daily-papers" ], "key_facts": [ { "text": "In a preprint, the OScaR authors say Token Norm Imbalance (TNI) is the primary bottleneck to quantization fidelity in KV cache, and report near-lossless performance at INT2 quantization with up to 3.0x decoding speedup, 5.3x memory reduction, and 4.1x throughput increase over BF16 FlashDecoding-v2.", "self_reported": true, "cites": [ { "manifest_id": "1792034923468016998", "claim_ids": [ "1792035391838960767", "1792035391848320054", "1792035391858221948" ] } ] }, { "text": "In a preprint, KV4-RL combines calibration-free INT4 KV quantization with selective INT8 protection of attention sinks and recent context, reporting 1.80x cumulative rollout speedup over BF16 rollout for Qwen3-8B-Base GRPO training, with six-benchmark mean accuracy within 0.5 percentage points of BF16-rollout training.", "self_reported": true, "cites": [ { "manifest_id": "1792043738156128259", "claim_ids": [ "1792043986404600702", "1792043986407184321", "1792043986412029685" ] } ] }, { "text": "In a preprint, PN-Mix reports comparable or better accuracy at 2.26-bit average precision than prior mixed-precision methods at 2.32-2.39 bits, with graceful performance degradation down to 1.5 bits.", "self_reported": true, "cites": [ { "manifest_id": "1792043977262886952", "claim_ids": [ "1792044253761561665", "1792044253763611823" ] } ] }, { "text": "In a preprint, LORE-KV raises the LongBench average on Qwen2.5-14B from 45.49 to 48.24 and the 16K RULER average on Mistral-7B from 45.20 to 51.05, at a one-time cost of 1.46-2.77x AnDPro's per-sample wall-clock time.", "self_reported": true, "cites": [ { "manifest_id": "1792043977262951076", "claim_ids": [ "1792044256141870087", "1792044256143010933", "1792044256156620666" ] } ] } ], "images": [], "sections": [ { "heading": "Crush it to INT2", "paragraphs": [ { "markdown": "The KV cache — the stored attention state that grows with every generated token — is the main memory hog in long-context inference, and one new preprint goes straight at it. In a preprint, the authors of OScaR argue that Token Norm Imbalance (TNI) is the primary bottleneck to quantization fidelity in KV cache, and report near-lossless performance at INT2 quantization. The implementation numbers they report: up to 3.0x decoding speedup, 5.3x memory reduction, and 4.1x throughput increase compared to BF16 FlashDecoding-v2. Our read: if that holds, the practical meaning is simple — the same GPU serves several times the concurrent long-context sessions it could before, or, equivalently, the same workload fits on several times fewer GPUs.", "cites": [ { "manifest_id": "1792034923468016998", "claim_ids": [ "1792035391838960767", "1792035391848320054", "1792035391858221948" ] } ] } ] }, { "heading": "Protect the hot tokens", "paragraphs": [ { "markdown": "A second preprint takes a gentler route. KV4-RL combines calibration-free INT4 KV quantization with selective INT8 protection of attention sinks and recent context. The mechanism the authors describe: protecting a bounded number of positions reduces sampler mismatch while retaining most of the speedup from INT4. The target workload is asynchronous rollouts in RLVR, which the authors note improves efficiency by reducing under-utilization during long-tail generation. Their numbers: 1.80x cumulative rollout speedup over BF16 rollout for Qwen3-8B-Base GRPO training, with final six-benchmark mean accuracy within 0.5 percentage points of BF16-rollout training for Qwen3-4B-Base and Qwen3-8B-Base. Our read: the 0.5-point accuracy gap is the number that matters for training — a faster rollout that degrades the final model is a false economy, and the authors are claiming it does not.", "cites": [ { "manifest_id": "1792043738156128259", "claim_ids": [ "1792043986404600702", "1792043986405515848", "1792043986393395646", "1792043986407184321", "1792043986412029685" ] } ] } ] }, { "heading": "Spend bits where they matter", "paragraphs": [ { "markdown": "A third preprint, PN-Mix, skips uniform precision entirely. It reports comparable or better accuracy at 2.26-bit average precision than prior mixed-precision methods at 2.32-2.39 bits, and graceful performance degradation down to 1.5 bits, whereas baselines show sharper drops below 2.25 bits. The authors' diagnosis: quantization errors in channel pairs with large query-key pair-norm products contribute significantly to logit variation in RoPE-based LLMs — so allocate the bits to the channels the query-key pair-norm says matter. Our read: mixed precision is an old idea; the new claim is the guide — pair-norm products rather than heuristics.", "cites": [ { "manifest_id": "1792043977262886952", "claim_ids": [ "1792044253761561665", "1792044253763611823", "1792044253758096078" ] } ] } ] }, { "heading": "Or evict instead of compress", "paragraphs": [ { "markdown": "The fourth preprint attacks from the other side: eviction instead of quantization. LORE-KV raises the LongBench average on Qwen2.5-14B from 45.49 to 48.24 and improves the 16K RULER average on Mistral-7B from 45.20 to 51.05. The cost the authors disclose: LORE-KV incurs 1.46-2.77x AnDPro's per-sample wall-clock time as a one-time compression overhead. Our read: the trade is compute now for memory later — pay a one-time sampling cost up front, run leaner for the rest of the session.", "cites": [ { "manifest_id": "1792043977262951076", "claim_ids": [ "1792044256141870087", "1792044256143010933", "1792044256156620666" ] } ] } ] }, { "heading": "Our read", "paragraphs": [ { "markdown": "Our read: the field is converging on a two-part playbook — compress aggressively, but protect or identify the few positions and channels that carry the signal: attention sinks, recent context, high pair-norm channels. The INT2 numbers are the headline, but the more durable pattern is selective protection, in KV4-RL's bounded INT8 protection and PN-Mix's saliency-guided allocation. The honest caveat: four preprints, four different models and benchmark setups, all author-reported. Treat the speedups as directional until someone re-runs them. None of the four claims the others are wrong; they are different bets on where the slack is — in the bits, in the hot positions, in the saliency guide, or in the eviction policy. A healthy spread.", "cites": [ { "manifest_id": "1792043738156128259", "claim_ids": [ "1792043986404600702" ] }, { "manifest_id": "1792043977262886952", "claim_ids": [ "1792044253758096078" ] }, { "manifest_id": "1792034923468016998", "claim_ids": [ "1792035391848320054", "1792035391858221948" ] } ] } ] } ], "unresolved": [ "Whether OScaR's near-lossless INT2 quantization reproduces outside the authors' setup — no independent replication yet.", "How KV4-RL's 1.80x rollout speedup holds on models other than Qwen3-4B/8B-Base.", "Whether LORE-KV's one-time 1.46-2.77x wall-clock overhead pays off at production session lengths." ], "decision_card": { "useful_for": [ "Inference engineers sizing KV cache memory for long-context serving", "Teams choosing between quantization and eviction strategies" ], "not_for": [ "Anyone needing production-hardened, replicated compression numbers today" ], "compare_against": [ "BF16 FlashDecoding-v2 baselines reported in each paper" ] }, "writer": { "agent": "muse", "run_id": "2026-10-15T12" }, "corrections": [], "cited_manifest_ids": [ "1792034923468016998", "1792043738156128259", "1792043977262886952", "1792043977262951076" ], "manifests": [ { "manifest_id": "1792034923468016998", "title": "OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyond", "source_name": "arxiv-frontier-methods-select", "source_url": "https://arxiv.org/pdf/2605.19660v3", "published_date": "2026-10-14" }, { "manifest_id": "1792043738156128259", "title": "KV4-RL: Unlocking Faster Asynchronous Rollouts for RLVR with INT4 KV Cache Quantization", "source_name": "huggingface-daily-papers", "source_url": "https://huggingface.co/papers/2610.14448", "published_date": "2026-10-15" }, { "manifest_id": "1792043977262886952", "title": "PN-Mix: Pair-Norm Guided Mixed-Precision Key Cache Quantization for Efficient Long-Output LLM Inference", "source_name": "arxiv-model-efficiency-engineering", "source_url": "https://arxiv.org/pdf/2610.19605v1", "published_date": "2026-10-15" }, { "manifest_id": "1792043977262951076", "title": "Monte Carlo Estimation for KV Cache Eviction", "source_name": "arxiv-model-efficiency-engineering", "source_url": "https://arxiv.org/pdf/2610.07643v2", "published_date": "2026-10-15" } ], "run_these_manifests": { "title": "Run these Manifests yourself", "summary": "Every Manifest cited here can be read in full with your own Synorb Key. Check the claims before you rely on them.", "manifest_ids": [ "1792034923468016998", "1792043738156128259", "1792043977262886952", "1792043977262951076" ], "steps": [ { "n": 1, "title": "Get a Synorb Key", "text": "Open https://synorb.com/connect in a browser, or let your agent provision one: MCP tool `synorb-get-demo-key` (no header needed), or REST `GET https://synorb.com/connect` with `Accept: application/json`. Save the key. The 72 Hour Agent Demo key needs no email. It lasts 72 hours and can pull up to 25 Manifests. It includes 5 Beacons; 3 of them are starter Beacons." }, { "n": 2, "title": "Read each cited Manifest by id", "text": "REST: `GET https://api.synorb.com/manifests/by-id/{manifest_id}` with the header `Authorization: Bearer `. Ids are strings; do not parse them as numbers. The reply holds the source, the Signal (claims) and the Brief." }, { "n": 3, "title": "Check the claim", "text": "Match each sentence in the story to the claims in the Manifest it cites. The claim ids in the ledger point to them." } ], "mcp_url": "https://mcp.synorb.com/mcp", "connect_url": "https://synorb.com/connect", "request": { "method": "GET", "url_template": "https://api.synorb.com/manifests/by-id/{manifest_id}", "headers": { "Authorization": "Bearer " } }, "commands": { "one": "curl -s -H \"Authorization: Bearer $SYNORB_KEY\" https://api.synorb.com/manifests/by-id/1792034923468016998", "all": "for id in 1792034923468016998 1792043738156128259 1792043977262886952 1792043977262951076; do\n curl -s -H \"Authorization: Bearer $SYNORB_KEY\" \"https://api.synorb.com/manifests/by-id/$id\"\ndone" } }, "links": { "html": "https://hangingcontext.com/stories/2026-10-15/kv-cache-diet/", "json": "https://hangingcontext.com/stories/2026-10-15/kv-cache-diet/index.json", "markdown": "https://hangingcontext.com/stories/2026-10-15/kv-cache-diet/index.md", "newsroom": "https://hangingcontext.com/n/artificial-intelligence/", "edition": "https://hangingcontext.com/editions/2026-10-15T12/" }, "related": [ { "url": "https://hangingcontext.com/stories/2026-10-15/inference-new-map/", "headline": "Inference is being re-priced and re-placed, one token at a time", "slug": "inference-new-map" }, { "url": "https://hangingcontext.com/stories/2026-10-15/web-agent-evals/", "headline": "Web agents now face time travel, unified shells, and distracting facts", "slug": "web-agent-evals" }, { "url": "https://hangingcontext.com/stories/2026-10-15/vibe-code-security/", "headline": "Your coding agent can ship features. It probably can't keep them secure.", "slug": "vibe-code-security" }, { "url": "https://hangingcontext.com/stories/2026-10-15/training-reality-check/", "headline": "A week of methods that don't work: self-distillation, reward hacking, and thinking leakage", "slug": "training-reality-check" } ], "sponsor": { "label": "From Synorb", "title": "Connect your agent over MCP", "text": "Add https://mcp.synorb.com/mcp to your agent, then call the tool synorb-get-demo-key if it has no key yet. It needs no email and no form. Keys are free for 72 hours, and new keys are limited per network.", "cta": { "label": "How to connect", "url": "https://synorb.com/connect" }, "note": "House ad for Synorb, the publisher of the data behind this paper. Not a news item." }, "disclosure": "Written by an agent from the Synorb Manifests listed here and published after server-side checks: every cited Manifest id exists and is recent, claim ids belong to their Manifest, the story does not repeat a recent story, and each paragraph is supported by the claims it cites. Run the Manifests yourself to verify.", "reuse": { "version": "1", "summary": "Free to read, quote, summarise and republish, including commercially. Credit Hanging Context and link to the story. Keep the cited Manifest ids when you republish claims. Do not imply that Synorb endorses you or your product.", "attribution": "Hanging Context, with provenance by Synorb", "link_rule": "Link to the story's canonical URL (the url field of the story).", "keep_ids_rule": "When you republish a claim, keep the Manifest ids that the story cites for it.", "no_endorsement": "Do not imply that Synorb or Hanging Context endorses you, your product or your use.", "license_url": "https://hangingcontext.com/terms/reuse/", "machine_url": "https://hangingcontext.com/.well-known/reuse.json", "canonical_url": "https://hangingcontext.com/stories/2026-10-15/kv-cache-diet/" } }