Smaller without the cliff: compression that respects the model's structure
PrismQuant aligns outliers for low-bit quantization, GLUESTICK recovers pruned robot models without retraining, and LayerTrim prunes MLPs while keeping attention intact.
- In a preprint, PrismQuant reports state-of-the-art W4A4KV4 performance on Llama-3.2-3B, and 3.85 perplexity with 72.46% zero-shot accuracy on Llama-3.1-70B — within 0.22 percentage points of FP16. Self-reported m:…2405
- In the same preprint, PrismQuant reports 1.51x prefill and 1.22x CUDA Graph decode speedups plus a 56.34% reduction in decode peak memory on Llama-3.1-8B. Self-reported m:…2405
- In a preprint, naive pruning of vision-language-action models is reported to cause drastic performance degradation and increased safety violations, while GLUESTICK restores up to 100% of success rates and near-dense safety levels post-pruning — with up to 5 GB memory savings and 1.75x speedup, without additional training. Self-reported m:…7303
- In a preprint, LayerTrim — training-free structured pruning that trims MLP features while keeping attention and decoder depth — reports 1.24x faster prefill and 19.7% lower peak GPU memory on Qwen3-14B, improving average accuracy by up to 21.4 percentage points over the RESP pruning method. Self-reported m:…7725
Rotate the outliers away
Low-bit quantization lives or dies on outliers: a few large activations wreck the quantization grid for everything else. In a preprint, the authors of PrismQuant address this with a quantizer-aware rotation framework that aligns activation outliers with the constant group subspace of asymmetric grouped INT4. The reported results: state-of-the-art W4A4KV4 performance on Llama-3.2-3B, and on Llama-3.1-70B a perplexity of 3.85 with 72.46% zero-shot accuracy — within 0.22 percentage points of full precision. On Llama-3.1-8B the practical payoff is reported as 1.51x faster prefill, 1.22x faster CUDA Graph decode, and 56.34% lower decode peak memory. m:…2405
The 70B result is the one that matters: near-FP16 accuracy at 4-bit weights, activations, and KV cache on a model that size says low-bit quantization has moved from 'acceptable degradation' to 'measure twice before you notice.' The rotation idea is doing the work — instead of asking the quantizer to tolerate outliers, reshape the space so the outliers stop being outliers. This suggests teams still serving 70B models in FP16 for quality reasons should re-run the comparison; the quality gap the industry priced in may have closed faster than the deployment defaults were updated. m:…2405
Pruning breaks robots; glue fixes them
Pruning is riskier when the model moves a physical arm. In a preprint with the memorable title 'Don't Run with Scissors,' the authors report that naive pruning of vision-language-action models causes drastic performance degradation and increased safety violations — the compressed model does not just get worse, it gets dangerous. Their fix, GLUESTICK, is a post-pruning recovery method based on weight-space interpolation between the dense and pruned models. It is reported to restore up to 100% of the success rate and bring safety back to near-dense levels, delivering up to 5 GB of memory savings and 1.75x speedup without any additional training. m:…7303
The safety-violation finding is the part that should travel beyond the robotics niche. Pruning is usually evaluated on accuracy benchmarks; this paper says the failure mode that matters for VLA models is safety, and that it degrades faster than accuracy does. GLUESTICK's interpolation approach suggests the pruned model retains the right structure but loses calibration — a fixable problem, and fixable without retraining, which is what makes it deployable on edge hardware where retraining is not an option. The implication for anyone pruning models that act in the world: measure safety, not just accuracy, and budget for a recovery step. m:…7303
Keep the attention, trim the MLPs
The third preprint brings the same structural respect to reasoning models. LayerTrim is a training-free structured pruning method that trims redundant MLP features while deliberately keeping attention and decoder depth intact — the bet being that reasoning capability lives more in the attention machinery than in MLP width. On Qwen3-14B the reported payoff is 1.24x faster prefill and 19.7% lower peak GPU memory versus the dense model, and the method is reported to improve average accuracy by up to 21.4 percentage points over the RESP structured pruning baseline. m:…7725
Across all three preprints, the lesson is the same: compression works when it respects what the model is doing. Rotate the activation space so outliers align with the quantizer; interpolate back toward the dense weights when pruning breaks a robot's safety; trim the MLPs but leave the attention alone when the model needs to reason. Blind sparsity is a cliff — 100% of success rate lost, safety violations gained — and structure-aware compression is the guardrail. For inference teams, the practical takeaway is to treat the compression method as a first-class choice alongside the model: the reported numbers suggest the gap between the best and worst way to shrink a model is now larger than the gap between model sizes. m:…2405 m:…7303 m:…7725
What is still unresolved
- All three papers are preprints; the reported accuracy, speedup, and recovery figures are the authors' own measurements.
- PrismQuant's near-FP16 claim is measured on zero-shot accuracy and perplexity; task-specific degradation is not covered by the cited claims.
- GLUESTICK was evaluated on VLA models; whether interpolation recovery generalizes to pure language models is not claimed.
Provenance by Synorb. Every claim links to its source. This story was written by an agent from the Manifests listed below. Read the methodology and limitations, or send a correction.
Free to reuse, including commercially. Credit “Hanging Context, with provenance by Synorb” and link to this story. Keep the cited Manifest ids. Reuse terms v1.