Quantization's playbook is splitting: INT4 tricks fail on new formats while InfoQuant and QSelect push 4-bit and 3-bit harder

The INT4 recipe breaks on new formats

In a preprint, the authors ran a systematic comparison of post-training quantization methods for large language models, splitting the pipeline into pre-quantization transformation, which reduces outliers before rounding, and error mitigation, which compensates for the performance lost to rounding. The point of the exercise is practical: give engineers a decision framework for picking a quantization strategy instead of cargo-culting last year's recipe. The headline result is that the recipe does not carry over. Optimizing rotation and scaling gives the best pre-quantization transformation for INT4, the authors report, but rotation-based transformations show little improvement on newer formats such as MXFP4 and NVFP4. m:…9146

Two more findings from the same evaluation complicate the usual rules of thumb. Combining low-rank compensation with GPTQ occasionally outperforms GPTQ on its own, which suggests the hybrid is worth testing on specific architectures rather than assumed to win everywhere. And asymmetric quantization benefits activations significantly more than it benefits weights, which points the optimization effort at activation layers when accuracy retention is the binding constraint. m:…9146

InfoQuant: shape the activations, keep 97% of accuracy

A second preprint takes the problem from the other side. InfoQuant treats activation transformation as a distribution design problem: quantization-friendly activations need both a smaller numerical range and enough dispersion within that range. Its mechanism, Peak Suppression Orthogonal Transformation (PSOT), reshapes activation distributions to be more quantization-friendly, and the authors report that it improves low-bit activation quantization by doing so. m:…2210

The numbers the authors report are strong for aggressive quantization: 97% of floating-point accuracy preserved at W4A4KV4, with weights, activations and KV cache all at 4 bits, and a 42% reduction in the performance gap on LLaMA-2 13B compared with previous state-of-the-art post-training quantization methods. These are the authors' own measurements from their evaluation, not independent verification, so read them as a claim to reproduce rather than a settled result. m:…2210

QSelect: stop picking one quantizer

A third preprint, introducing QSelect, rejects the premise that you should commit to a single quantizer family at all. The method composes a static low-bit model module by module, assigning each linear module to whichever of the AWQ, GPTQ or AutoRound quantizers scores best on next-token distribution. On held-out perplexity, the mixed model outperformed single-family quantization across DeepSeek-R1-Distill-Qwen-7B, Llama-3.1-8B and Llama-3.2-3B. The paper frames the problem as behavioral fidelity: the compressed model has to behave like its full-precision reference, so the selection criterion is what the model outputs, not what its internals look like. m:…5498

Two details matter for anyone shipping a quantized model. First, QSelect picks its module substitutions with final-output scoring, and the authors found that superior to representation-similarity scoring, the more common proxy. Judge the compression by what the model says. Second, the layers teams usually leave unquantized are not free: embedding and output layers account for 43 to 46 percent of the packed artifact size at 3-bit quantization. If your memory budget assumed those layers were rounding error, redo the math. m:…5498

What to try this week

Read together, the three preprints point in one direction: the era of a single default quantization recipe is ending. The PTQ evaluation warns that INT4 optimizations do not transfer to MXFP4 and NVFP4. InfoQuant argues the lever is the activation distribution, not the quantizer. QSelect argues the lever is mixing quantizers per module and scoring on outputs. The practical upshot for a team shipping a model this week: check which format your hardware actually runs, and do not assume the INT4 recipe you know still holds. m:…9146 m:…2210 m:…5498