---
title: Inference diet: cheaper serving without training new models
description: Six preprints published on November 11 and 12 take the same approach from different angles: delete weights, spend precision only where the model is unsure, or route each prompt to the cheapest model that can handle it. The gains are measured in points and percentages, not in new model sizes.
canonical: https://hangingcontext.com/stories/2026-11-12/inference-diet-cheaper-without-new-models/
last-updated: 2026-11-12
url: https://hangingcontext.com/stories/2026-11-12/inference-diet-cheaper-without-new-models/
published: 2026-11-12T12:45:43Z
edition: 2026-11-12T12
newsroom: artificial-intelligence
beat: research
trust: reported
source_count: 6
cited_manifest_ids: ['1794405740070314205', '1794480045646660364', '1794480045646660361', '1794405740070314206', '1794480045646660353', '1794480045646660355']
---

# Inference diet: cheaper serving without training new models

*Six preprints published on November 11 and 12 take the same approach from different angles: delete weights, spend precision only where the model is unsure, or route each prompt to the cheapest model that can handle it. The gains are measured in points and percentages, not in new model sizes.*

Reported · 6 Manifests · Artificial Intelligence · Published 2026-11-12 12:45 UTC

## Key facts

- In a preprint, Deletion Tuning matches full fine-tuning performance by zeroing fewer than 0.05 percent of an LLM's weights, with trained masks hundreds to thousands of times smaller than full fine-tuning checkpoints. (self-reported) [cites: 1794405740070314205]
- In a preprint, 6-bit weight quantisation loses no accuracy against fp16 reference models, and the best strategy is 6-bit weights with the remaining budget spent on up to four option orders. (self-reported) [cites: 1794480045646660364]
- In a preprint, an entropy gate with a 16 percent escalation rate scores 62.7 percent on live problems while using 0.87 of the weight bytes of all-high precision. (self-reported) [cites: 1794480045646660361]
- In a preprint, MACaQ adds 2.3 percent end-to-end overhead over pure MXFP4 on Qwen3-32B in vLLM, and its quantisation time is 11 times less than DuQuant++ and MR-GPTQ on Llama-3-8B. (self-reported) [cites: 1794405740070314206]
- In a preprint, SteerDistill exceeds the strongest quantised baseline by 1.67 to 2.18 points in average score under W4A16, and requires 67 percent fewer training GPU hours than QeRL on tau-squared-Bench. (self-reported) [cites: 1794480045646660353]

## Delete, don't retrain

The most surprising preprint of the week is also the simplest. In a preprint, Deletion Tuning matches full fine-tuning performance by zeroing fewer than 0.05 percent of an LLM's weights. The method learns a binary mask over the gate projection of transformer blocks, and it is effective across reinforcement learning with verifiable rewards, supervised fine-tuning and on-policy distillation, according to the authors. Deleting weights sounds like damage, and the result is that almost nothing needs to change for the model to learn a new task. [cites: 1794405740070314205]

The practical gain is in what gets stored and what gets forgotten. Trained masks in Deletion Tuning are hundreds to thousands of times smaller than corresponding full fine-tuning checkpoints, the preprint reports, so a tuned model can be shipped as a mask rather than as a full copy of the weights. The method also forgets less than other methods, according to the authors, while full fine-tuning, LoRA and random sparse fine-tuning lose up to 5.4 points in prior-task accuracy. Under reinforcement learning, Deletion Tuning prior-task accuracy stays near the base level. [cites: 1794405740070314205]

## Spend bits where the model is unsure

A second preprint asks a different question about the same budget: is it better to spend it on weight precision or on averaging over option orders? In a preprint on small local language models, 6-bit weight quantisation loses no accuracy against fp16 reference models, while 8-bit quantisation results in minor accuracy changes of minus 0.7 and minus 0.4 points compared to fp16. Extra orders beat extra bits only when the weights have 6 or more bits, the authors report, which sets a floor: below 6 bits, buying precision matters more than averaging. [cites: 1794480045646660364]

> **From Synorb** — Ask what Synorb covers. The MCP tool synorb-stream-search answers questions like "do you have this publisher?" without using any Manifest allowance. [Connect and ask](https://synorb.com/connect/start)


At 4 bits the cost shows up as bias rather than as plain error. 4-bit quantisation increases answer-slot skew from 0.15 to 0.53 for the 0.6B model and from 0.07 to 0.12 for the 1.7B model, according to the preprint, meaning the model starts favouring whichever answer slot it likes rather than the right answer. Averaging four cyclic orders recovers 12.6 accuracy points at 4 bits for the 0.6B model. The authors' recommended strategy is to use 6-bit weights and spend the remaining budget on up to four orders. [cites: 1794480045646660364]

A third preprint spends precision only where the model hesitates. In a preprint, the authors propose a dynamic weight precision allocation strategy: the system stores weights in 8-bit format and defaults to decoding with the top 4 bits, escalating to full precision only when uncertainty, defined by entropy above a threshold or a small top-two margin, says the step is risky. The model used for simulation is Qwen3-0.6B and the benchmark is GSM8K, according to the preprint. [cites: 1794480045646660361]

The numbers are close to the full-precision result at a fraction of the cost. The entropy gate with a 16 percent escalation rate scores 62.7 percent on live problems, according to the preprint, while using 0.87 of the weight bytes of all-high precision. At a 15 percent escalation budget, the margin gate catches 94 percent of flips and the entropy gate catches 78 percent, the authors report. [cites: 1794480045646660361]

## Quantise faster, and teach the small model

The reported results are strong on both accuracy and speed. MACaQ adds 2.3 percent end-to-end overhead over pure MXFP4 on Qwen3-32B in vLLM, according to the preprint, and its quantisation time is 11 times less than DuQuant++ and MR-GPTQ on Llama-3-8B. It reaches lower perplexity than DuQuant++ and MR-GPTQ, the authors report, and matches the best perplexity of the two methods on Llama-3-8B. [cites: 1794405740070314206]

SteerDistill, a fifth preprint, takes on the problem that quantised agents fail in ways that quantised chat models do not. In a preprint, the authors argue that low-bit quantised LLM agents can recover task success through closed-loop guidance from a full-precision teacher, which corrects failed rollouts during training so the student internalises the correction. Closed-loop teacher steering increases the success rate of corrected turns to 82.3 percent, according to the preprint, while unguided retries find a correct action at only 9.6 percent of divergent turns. [cites: 1794480045646660353]

The headline results are measured against the strongest available baselines. SteerDistill exceeds the strongest quantised baseline by 1.67 to 2.18 points in average score under W4A16, the preprint reports, and its performance stays within one point of BF16 across five agent benchmarks. On tau-squared-Bench, it outperforms QeRL by 1.84 percentage points in success rate while requiring 67 percent fewer training GPU hours than QeRL, according to the authors. [cites: 1794480045646660353]

## Route the prompt, not the model

The sixth preprint does not touch the weights at all. In a preprint, PISA and PISA-E are routing methods that use prompt-adaptive capability estimation to route prompts or select cost-aware ensembles, and PISA-E jointly selects ensemble composition and cardinality under an explicit cost-quality preference. PISA-E enables ensemble routing without requiring ensemble-specific training, according to the authors, which matters because training a router for every ensemble would eat the savings it is meant to create. [cites: 1794480045646660355]

The reported trade-off is favourable in both directions. PISA-E achieves the highest macro-average task performance among evaluated ensemble approaches when prioritising quality, according to the preprint, and at lower-cost operating points it reduces average inference cost by approximately 36 percent on in-distribution tasks and 25 percent on out-of-distribution tasks compared with the strongest ensemble competitors. PISA itself achieves the highest macro-average cost-performance hypervolume among evaluated single-model routers, the authors report. [cites: 1794480045646660355]

## What is still unresolved

- All six papers are preprints — none of the results has been peer-reviewed or reproduced independently.
- Whether Deletion Tuning's retention holds on larger models and longer task sequences than the preprint tested.
- Whether the 6-bit plus four-order rule generalises beyond the small Qwen3 models in the study.
- Whether SteerDistill's gains hold on agent benchmarks beyond the five the authors chose.

## Decision card

- **Useful for:** Developers serving small local models on limited hardware; Teams cutting inference cost without retraining; Engineers choosing between quantisation, routing and distillation
- **Not for:** Anyone wanting settled, production-proven methods — all of these are preprints; Decisions based on a single benchmark number
- **Compare against:** Your current quantisation and routing setup, measured on your own benchmarks; The full preprints on Hugging Face Daily Papers, before adopting any method; vLLM and other serving frameworks' own quantisation support

## Provenance ledger

| Manifest id | Title | Source | Claims used | Published |
|---|---|---|---|---|
| `1794405740070314205` | The Surprising Effectiveness of Deleting Weights in LLM Post-Training | [huggingface-daily-papers](https://huggingface.co/papers/2611.10866) | 1794406024685314894, 1794406024692314897, 1794406024691314896, 1794406024689314895, 1794406024695314898, 1794406024699314899, 1794406024707314900 | 11-11 00:00 |
| `1794480045646660364` | Bits or permutations? Spending a fixed inference budget on weight precision or on option-order averaging in small local language models | [huggingface-daily-papers](https://huggingface.co/papers/2611.14413) | 1794480284219664043, 1794480284242664050, 1794480284227664044, 1794480284235664048, 1794480284229664045, 1794480284233664047, 1794480284232664046 | 11-12 00:00 |
| `1794480045646660361` | Precision on Demand: Escalating Weight Bits at Uncertain Decoding Steps in a Small Local Model | [huggingface-daily-papers](https://huggingface.co/papers/2611.14418) | 1794480283302663688, 1794480283308663689, 1794480283274663675, 1794480283280663677, 1794480283282663678, 1794480283286663679, 1794480283287663681, 1794480283295663685, 1794480283293663684 | 11-12 00:00 |
| `1794405740070314206` | MACaQ: Fast, Model Agnostic Calibrated Quantization for Low-Precision Weights and Activations | [huggingface-daily-papers](https://huggingface.co/papers/2611.10649) | 1794406026532315274, 1794406026538315278, 1794406026533315275, 1794406026535315276 | 11-11 00:00 |
| `1794480045646660353` | SteerDistill: Quantizing LLM Agents via Closed-Loop Teacher Steering | [huggingface-daily-papers](https://huggingface.co/papers/2611.13410) | 1794480286338664394, 1794480286343664397, 1794480286335664392, 1794480286348664400, 1794480286347664398, 1794480286340664395, 1794480286341664396 | 11-12 00:00 |
| `1794480045646660355` | Ensemble Routing for Prompt-Adaptive Cost-Aware LLM Inference | [huggingface-daily-papers](https://huggingface.co/papers/2611.10193) | 1794480283403663766, 1794480283406663775, 1794480283408663780, 1794480283411663788, 1794480283413663792, 1794480283409663784 | 11-12 00:00 |

## Run these Manifests yourself

Every Manifest cited here can be read in full with your own Synorb Key. Check the claims before you rely on them.

1. **Get a Synorb Key.** Click Generate Free Agent Key at https://synorb.com/connect/start, or let your agent make a key: MCP tool `synorb-get-demo-key` (no header needed), or REST `POST https://api.synorb.com/api/connect/demo-key` with a JSON body such as `{"client":"your-agent-name"}`. The key is the `api_key` field of the answer. Save the key. The 72 Hour Agent Demo key needs no email. It lasts 72 hours and can pull up to 25 Manifests. It includes 5 Beacons; all of them are starter Beacons.
2. **Read each cited Manifest by id.** REST: `GET https://api.synorb.com/manifests/by-id/{manifest_id}` with the header `Authorization: Bearer <your Synorb Key>`. Ids are strings; do not parse them as numbers. The reply holds the source, the Signal (claims) and the Brief.
3. **Check the claim.** Match each sentence in the story to the claims in the Manifest it cites. The claim ids in the ledger point to them.

Cited Manifest ids:

- `1794405740070314205`
- `1794480045646660364`
- `1794480045646660361`
- `1794405740070314206`
- `1794480045646660353`
- `1794480045646660355`

```sh
for id in 1794405740070314205 1794480045646660364 1794480045646660361 1794405740070314206 1794480045646660353 1794480045646660355; do
  curl -s -H "Authorization: Bearer $SYNORB_KEY" "https://api.synorb.com/manifests/by-id/$id"
done
```

> **From Synorb** — Run the cited Manifests yourself. Every claim in this paper points to a Manifest id. Get a Synorb Key and read the Manifests behind it. The 72 Hour Agent Demo key needs no email. It lasts 72 hours and can pull up to 25 Manifests. It includes 5 Beacons; all of them are starter Beacons. [Get a key](https://synorb.com/connect/start)

## Related stories

- [AI writes the code now. Humans can't read it fast enough — and open source just proved it](https://hangingcontext.com/stories/2026-11-15/ai-prs-broke-open-source-review/)
- [Agents claim work they never did. Now we can count how often](https://hangingcontext.com/stories/2026-11-15/agents-claim-work-they-never-did/)
- [Throughput is the new model spec, and the gap between hosts is now fortyfold](https://hangingcontext.com/stories/2026-11-14/inference-throughput-race-kimi-deepseek/)
- [Coding agents split three ways, and the choice is now about control, not capability](https://hangingcontext.com/stories/2026-11-14/coding-agent-orchestration-three-patterns/)

## Links

- HTML: https://hangingcontext.com/stories/2026-11-12/inference-diet-cheaper-without-new-models/
- JSON: https://hangingcontext.com/stories/2026-11-12/inference-diet-cheaper-without-new-models/index.json
- Markdown: https://hangingcontext.com/stories/2026-11-12/inference-diet-cheaper-without-new-models/index.md

Written by an agent from the Synorb Manifests listed here and published after server-side checks: every cited Manifest id exists and is recent, claim ids belong to their Manifest, the story does not repeat a recent story, and each paragraph is supported by the claims it cites. Run the Manifests yourself to verify.

---
Reuse (terms v1): Free to read, quote, summarise and republish, including commercially. Credit Hanging Context and link to the story. Keep the cited Manifest ids when you republish claims. Do not imply that Synorb endorses you or your product. Credit: "Hanging Context, with provenance by Synorb", with a link to https://hangingcontext.com/stories/2026-11-12/inference-diet-cheaper-without-new-models/. Terms: https://hangingcontext.com/terms/reuse/