The KV cache goes on a diet: INT2 quantization nears lossless

Crush it to INT2

The KV cache — the stored attention state that grows with every generated token — is the main memory hog in long-context inference, and one new preprint goes straight at it. In a preprint, the authors of OScaR argue that Token Norm Imbalance (TNI) is the primary bottleneck to quantization fidelity in KV cache, and report near-lossless performance at INT2 quantization. The implementation numbers they report: up to 3.0x decoding speedup, 5.3x memory reduction, and 4.1x throughput increase compared to BF16 FlashDecoding-v2. Our read: if that holds, the practical meaning is simple — the same GPU serves several times the concurrent long-context sessions it could before, or, equivalently, the same workload fits on several times fewer GPUs. m:…6998

Protect the hot tokens

A second preprint takes a gentler route. KV4-RL combines calibration-free INT4 KV quantization with selective INT8 protection of attention sinks and recent context. The mechanism the authors describe: protecting a bounded number of positions reduces sampler mismatch while retaining most of the speedup from INT4. The target workload is asynchronous rollouts in RLVR, which the authors note improves efficiency by reducing under-utilization during long-tail generation. Their numbers: 1.80x cumulative rollout speedup over BF16 rollout for Qwen3-8B-Base GRPO training, with final six-benchmark mean accuracy within 0.5 percentage points of BF16-rollout training for Qwen3-4B-Base and Qwen3-8B-Base. Our read: the 0.5-point accuracy gap is the number that matters for training — a faster rollout that degrades the final model is a false economy, and the authors are claiming it does not. m:…8259

Spend bits where they matter

A third preprint, PN-Mix, skips uniform precision entirely. It reports comparable or better accuracy at 2.26-bit average precision than prior mixed-precision methods at 2.32-2.39 bits, and graceful performance degradation down to 1.5 bits, whereas baselines show sharper drops below 2.25 bits. The authors' diagnosis: quantization errors in channel pairs with large query-key pair-norm products contribute significantly to logit variation in RoPE-based LLMs — so allocate the bits to the channels the query-key pair-norm says matter. Our read: mixed precision is an old idea; the new claim is the guide — pair-norm products rather than heuristics. m:…6952

Or evict instead of compress

The fourth preprint attacks from the other side: eviction instead of quantization. LORE-KV raises the LongBench average on Qwen2.5-14B from 45.49 to 48.24 and improves the 16K RULER average on Mistral-7B from 45.20 to 51.05. The cost the authors disclose: LORE-KV incurs 1.46-2.77x AnDPro's per-sample wall-clock time as a one-time compression overhead. Our read: the trade is compute now for memory later — pay a one-time sampling cost up front, run leaner for the rest of the session. m:…1076

Our read

Our read: the field is converging on a two-part playbook — compress aggressively, but protect or identify the few positions and channels that carry the signal: attention sinks, recent context, high pair-norm channels. The INT2 numbers are the headline, but the more durable pattern is selective protection, in KV4-RL's bounded INT8 protection and PN-Mix's saliency-guided allocation. The honest caveat: four preprints, four different models and benchmark setups, all author-reported. Treat the speedups as directional until someone re-runs them. None of the four claims the others are wrong; they are different bets on where the slack is — in the bits, in the hot positions, in the saliency guide, or in the eviction policy. A healthy spread. m:…8259 m:…6952 m:…6998