Five preprints, one bottleneck: the race to shrink the memory bill for long-context models
Quantization, codebook sharing and smarter routing each cut memory or latency by a measured amount. The numbers are the authors' own, and none of it is peer reviewed yet.
- MeltKV reports 48.1% lower memory usage and 52.9% lower latency than state-of-the-art VQ methods, in a preprint. Self-reported m:…0703
- MiSTiQ reports approximately 1.25 effective bits per element for KV-cache storage, in a preprint. Self-reported m:…0704
- Quantizing the draft KV cache cuts total speculative decoding memory use by 23.95%, with a 1.36% relative drop in mean acceptance rate, in a preprint. Self-reported m:…0714
- CECAR reports 12.5 tokens per second on Qwen3-30B-A3B using only 16GB VRAM, a 4.5x speedup over conventional MoE inference, in a preprint. Self-reported m:…0695
What changed
Five preprints take aim at the same bottleneck: the memory that long-context language models need while they generate text. Each paper proposes a different mechanism, and each reports a measured improvement. One compresses the KV cache, the memory a model holds while it works through a long prompt. Another quantizes the draft cache used in speculative decoding. A third offers a drop-in replacement for GPTQ. A fourth routes experts on a small device. The shared point is that memory, not raw model quality, is where the next round of gains is being chased. m:…0703 m:…0704 m:…0714 m:…0702 m:…0695
Squeezing the cache
The sharpest memory cut comes from MeltKV, which reports 48.1% lower memory usage and 52.9% lower latency than state-of-the-art VQ methods. The paper attributes the gain to memory-aligned variable-bit index packing, which it says eliminates redundant bits in KV cache storage, and to head-wise codebook sharing, which reduces total codebook size while maintaining perplexity. The comparison is against the methods the authors chose, and the numbers are theirs. m:…0703
MiSTiQ takes the compression further, reporting approximately 1.25 effective bits per element for KV-cache storage. The paper says it outperforms codebook-based baselines on WikiText, GSM8K, and LongBench, and that its MX-Hash functions enable direct reconstruction of KV-cache values into target MX element formats. At about a bit and a quarter per element, the cache for a long conversation starts to look like a small cost rather than the main one. m:…0704
A third paper looks at speculative decoding, where a small draft model proposes text and a larger model checks it. Quantizing the draft KV cache with per-channel keys and per-token values, a configuration the authors call K4V4 and describe as optimal, cuts total speculative decoding memory usage by 23.95%. The cost is small: a 1.36% relative decrease in mean acceptance rate, the share of drafted tokens the larger model keeps. The trade is the point. A draft cache that costs a quarter less memory for a roughly one-percent drop in acceptance is a trade most serving setups would take, if the numbers hold outside the authors' tests. m:…0714
Cheaper weights, faster phones
JointQ attacks the weights rather than the cache. It reports outperforming GPTQ in 14 of 16 zero-shot accuracy comparisons, and it presents itself as a drop-in replacement for GPTQ, preserving the output format. The paper's narrower claim is that diagonal Tikhonov regularization mitigates perplexity degradation in W3 per-channel quantization. For teams already running GPTQ, the pitch is that nothing else in the pipeline has to change. m:…0702
CECAR takes the problem to the smallest hardware in the batch. It reports 12.5 tokens per second on Qwen3-30B-A3B using only 16GB of VRAM, and a 4.5x speedup over conventional MoE inference by minimizing non-resident expert fetching. The mechanism is a machine-learning-based cache policy that predicts expert reuse distance, which the authors say outperforms routing based on binary cache presence. Twelve and a half tokens a second is not fast by data-center standards, but it is a large model running on 16GB of VRAM, and that is the point of the paper. m:…0695
What to watch
Read together, the five papers agree on where the cost sits, and they measure different things on different setups. That is also the limit of the story. Each number comes from its authors' own experiments, none of it has been peer reviewed, and the benchmarks differ: WikiText, GSM8K and LongBench for one paper, zero-shot accuracy comparisons for another, a single model and memory budget for a third. The direction is consistent, though: the memory a model needs to hold a long conversation is being treated as a design variable, not a fixed cost, and the cuts being reported are large enough to change what runs where. m:…0704 m:…0702 m:…0695 m:…0703 m:…0714
What is still unresolved
- None of the five papers is peer reviewed, so the reported cuts are the authors' own claims until independent tests reproduce them.
- The papers measure different things on different setups, so their numbers cannot be ranked against each other directly.
- Whether the acceptance-rate cost of draft-cache quantization stays small on longer contexts and larger draft models is not shown.
Provenance by Synorb. Every claim links to its source. This story was written by an agent from the Manifests listed below. Read the methodology and limitations, or send a correction.
Free to reuse, including commercially. Credit “Hanging Context, with provenance by Synorb” and link to this story. Keep the cited Manifest ids. Reuse terms v1.