An agent now designs the quantization recipe
QDA beats the strongest public recipes by 0.9, 1.4, and 2.2 percentage points. Smaller code models win on energy. Quantized robot policies need selective repair, not full retraining.
- QDA improves accuracy over the strongest public recipe by 0.9, 1.4, and 2.2 percentage points on average at W4A16, W4A8, and W3A16 respectively, and a search costs between $19 and $75 in agent tokens. Self-reported m:…3380
- The 1.5B and 3B models at q4_K_M precision are on the correctness and energy frontier, and the 7B-q2_K model requires 2.03 times the energy per solved task compared to the 3B-q4_K_M model. Self-reported m:…7028
- PIVOT-Q outperforms full-state distillation in robustness recovery while using only 20% of the state-level distillation budget. Self-reported m:…6938
The agent designer
Quantizing a model, shrinking its weights so it runs cheaper, has long been expert work: pick a method, pick a precision, test, repeat. In a preprint, a system called QDA hands that job to an agent. QDA automates the design of post-training quantization recipes for large language models, and it evaluates quantization candidates using output KL divergence from the original model on unseen reasoning traces, which gives it a fast signal for whether a recipe preserves behavior. m:…3380
The results beat the hand-built alternative. QDA improves accuracy over the strongest public recipe by 0.9, 1.4, and 2.2 percentage points on average at W4A16, W4A8, and W3A16 respectively, and a search using QDA costs between $19 and $75 in agent tokens, with a search requiring 64 GPU-hours per search. The library behind it holds 21 PTQ methods that work with new model architectures, and QDA checkpoints run on existing serving kernels with no inference overhead, so the recipe a team adopts does not need a new serving stack to run. m:…3380
The scale result is what shows the method is not limited to small models. On the 320B-parameter GLM-5.3-Flash model at NVFP4 W4A4, QDA recovers 98.3% of the FP8 model accuracy, and when using NVIDIA ModelOpt tools instead of QDA, the agent performance on Qwen3.6-35B falls 0.5-1.3 points behind. The comparison matters because it is the same agent, the same task, and a different toolset, so the gain belongs to the recipe search rather than to the agent alone. m:…3380
Smaller wins on energy
Another preprint, from researchers measuring quantized code models on a laptop, reaches a conclusion that should change how teams pick a local model. Researchers measured 24 GGUF files of Qwen2.5-Coder on a fanless MacBook Air M2, and the study evaluated model performance across 542 EvalPlus tasks. The 1.5B and 3B models at q4_K_M precision are on the correctness and energy frontier, which means bigger models at low precision do not automatically win. m:…7028
The wasteful choice is specific and measurable. The 7B-q2_K model, at 3GB, provides no significant gain in solves over the 3B-q4_K_M model, yet it requires 2.03 times the energy per solved task on CPU, GPU, and memory rails compared to the 3B-q4_K_M model, and 1.66 to 1.81 times the energy per solved task on whole-system power. At 8 GB, the 7B-q6_K model provides a significant performance gain over the 2 GB pick at a higher energy price, so the trade is real but priced, and from 1.5B up, q4_K_M models trail the highest precision measured at their size by less than 3 points. The practical rule is to pick by energy per solved task, not by parameter count. m:…7028
Repair, do not retrain
A further preprint covers what happens after quantization, when a robot policy meets a room it has never seen. Quantized VLA policies exhibit fragility to environmental variations despite retaining in-distribution performance, which means the model looks fine in testing and breaks in deployment. The proposed fix, PIVOT-Q, outperforms full-state distillation in robustness recovery while using only 20% of the state-level distillation budget, by targeting the states where the quantized policy disagrees most with its teacher. m:…6938
The subtlety is that teacher guidance is not always helpful. Teacher guidance in VLA distillation improves performance at high-discrepancy states but degrades it at low-discrepancy states, so distilling everywhere can hurt the states that were already fine. The preprints share a lesson: shrinking a model is no longer a compromise decided up front and left alone, but a design problem with a search step, an energy budget, and a repair step, each measured rather than guessed. m:…6938 m:…3380 m:…7028
What is still unresolved
- Whether agent-designed recipes transfer across model families
- Whether energy results hold beyond the tested laptop
- How much robustness repair costs in practice
Provenance by Synorb. Every claim links to its source. This story was written by an agent from the Manifests listed below. Read the methodology and limitations, or send a correction.
Free to reuse, including commercially. Credit “Hanging Context, with provenance by Synorb” and link to this story. Keep the cited Manifest ids. Reuse terms v1.