Inference diet: cheaper serving without training new models

Delete, don't retrain

The most surprising preprint of the week is also the simplest. In a preprint, Deletion Tuning matches full fine-tuning performance by zeroing fewer than 0.05 percent of an LLM's weights. The method learns a binary mask over the gate projection of transformer blocks, and it is effective across reinforcement learning with verifiable rewards, supervised fine-tuning and on-policy distillation, according to the authors. Deleting weights sounds like damage, and the result is that almost nothing needs to change for the model to learn a new task. m:…4205

The practical gain is in what gets stored and what gets forgotten. Trained masks in Deletion Tuning are hundreds to thousands of times smaller than corresponding full fine-tuning checkpoints, the preprint reports, so a tuned model can be shipped as a mask rather than as a full copy of the weights. The method also forgets less than other methods, according to the authors, while full fine-tuning, LoRA and random sparse fine-tuning lose up to 5.4 points in prior-task accuracy. Under reinforcement learning, Deletion Tuning prior-task accuracy stays near the base level. m:…4205

Spend bits where the model is unsure

A second preprint asks a different question about the same budget: is it better to spend it on weight precision or on averaging over option orders? In a preprint on small local language models, 6-bit weight quantisation loses no accuracy against fp16 reference models, while 8-bit quantisation results in minor accuracy changes of minus 0.7 and minus 0.4 points compared to fp16. Extra orders beat extra bits only when the weights have 6 or more bits, the authors report, which sets a floor: below 6 bits, buying precision matters more than averaging. m:…0364

At 4 bits the cost shows up as bias rather than as plain error. 4-bit quantisation increases answer-slot skew from 0.15 to 0.53 for the 0.6B model and from 0.07 to 0.12 for the 1.7B model, according to the preprint, meaning the model starts favouring whichever answer slot it likes rather than the right answer. Averaging four cyclic orders recovers 12.6 accuracy points at 4 bits for the 0.6B model. The authors' recommended strategy is to use 6-bit weights and spend the remaining budget on up to four orders. m:…0364

A third preprint spends precision only where the model hesitates. In a preprint, the authors propose a dynamic weight precision allocation strategy: the system stores weights in 8-bit format and defaults to decoding with the top 4 bits, escalating to full precision only when uncertainty, defined by entropy above a threshold or a small top-two margin, says the step is risky. The model used for simulation is Qwen3-0.6B and the benchmark is GSM8K, according to the preprint. m:…0361

The numbers are close to the full-precision result at a fraction of the cost. The entropy gate with a 16 percent escalation rate scores 62.7 percent on live problems, according to the preprint, while using 0.87 of the weight bytes of all-high precision. At a 15 percent escalation budget, the margin gate catches 94 percent of flips and the entropy gate catches 78 percent, the authors report. m:…0361

Quantise faster, and teach the small model

The reported results are strong on both accuracy and speed. MACaQ adds 2.3 percent end-to-end overhead over pure MXFP4 on Qwen3-32B in vLLM, according to the preprint, and its quantisation time is 11 times less than DuQuant++ and MR-GPTQ on Llama-3-8B. It reaches lower perplexity than DuQuant++ and MR-GPTQ, the authors report, and matches the best perplexity of the two methods on Llama-3-8B. m:…4206

SteerDistill, a fifth preprint, takes on the problem that quantised agents fail in ways that quantised chat models do not. In a preprint, the authors argue that low-bit quantised LLM agents can recover task success through closed-loop guidance from a full-precision teacher, which corrects failed rollouts during training so the student internalises the correction. Closed-loop teacher steering increases the success rate of corrected turns to 82.3 percent, according to the preprint, while unguided retries find a correct action at only 9.6 percent of divergent turns. m:…0353

The headline results are measured against the strongest available baselines. SteerDistill exceeds the strongest quantised baseline by 1.67 to 2.18 points in average score under W4A16, the preprint reports, and its performance stays within one point of BF16 across five agent benchmarks. On tau-squared-Bench, it outperforms QeRL by 1.84 percentage points in success rate while requiring 67 percent fewer training GPU hours than QeRL, according to the authors. m:…0353

Route the prompt, not the model

The sixth preprint does not touch the weights at all. In a preprint, PISA and PISA-E are routing methods that use prompt-adaptive capability estimation to route prompts or select cost-aware ensembles, and PISA-E jointly selects ensemble composition and cardinality under an explicit cost-quality preference. PISA-E enables ensemble routing without requiring ensemble-specific training, according to the authors, which matters because training a router for every ensemble would eat the savings it is meant to create. m:…0355

The reported trade-off is favourable in both directions. PISA-E achieves the highest macro-average task performance among evaluated ensemble approaches when prioritising quality, according to the preprint, and at lower-cost operating points it reduces average inference cost by approximately 36 percent on in-distribution tasks and 25 percent on out-of-distribution tasks compared with the strongest ensemble competitors. PISA itself achieves the highest macro-average cost-performance hypervolume among evaluated single-model routers, the authors report. m:…0355