Cheaper per token is not cheaper per task: the energy bill hiding inside inference
Four new preprints show that quantization, code optimization, and routing all change the energy cost of finished work in surprising ways, while serverless inference makes idle time free and everything else metered.
- In a preprint, 4-bit quantization generally saves energy per token, but whether it saves energy per completed task is task dependent, and quantization can lead to up to 3.5 times as much energy per completed task. Self-reported m:…2392
- In a preprint, LLM self-refinement pipelines consistently produce code that consumes more energy per problem than canonical human reference solutions, while limiting output tokens reduces energy consumption by up to 69%. Self-reported m:…2488
- In a preprint, CRE-Router improves accuracy over efficient models while maintaining performance comparable to strong models at a lower cost per token, with an escalation mechanism that catches errors pre-generation routing cannot predict. Self-reported m:…8293
- In a preprint, Accordion's middle inference budget captures 89% of the maximum performance gain using half as many additional layer executions. Self-reported m:…1570
- Fal says its serverless inference adds runners as requests arrive and removes them when traffic stops, down to zero by default, billed per second of runner lifetime. Self-reported m:…1782
The meter runs on lifetime
Fal's documentation for its serverless inference platform describes a simple billing model: fal adds runners as requests arrive and removes them when traffic stops, down to zero by default, billed per second of runner lifetime. Setup, idle time including keep-alive, running, draining, and terminating states are all billed, while pending, docker pull, and terminated states are not, and requests that fail with a 5xx error are not charged. m:…1782
The operational consequence is that idle capacity costs nothing only if it actually scales to zero. The default maximum concurrency limit is 2 runners, so any app that needs to scale has to set it explicitly, and runtime-tunable parameters persist across deploys unless reset. The meter rewards finishing fast and disappearing, not merely being cheap per token. m:…1782
Cheaper per token is not cheaper per task
A preprint measuring the energy cost of completing tasks with quantized language-model agents challenges a common assumption: that per-token energy savings translate directly to task-level efficiency. Across 29 agent tasks, the authors show that 4-bit precision generally saves energy per token, but whether it saves energy per completed task is task dependent and changes with the model and the execution budget. m:…2392
The reason is that quantization changes both how much the agent generates and whether it succeeds, leading to up to 3.5 times as much energy per completed task in the worst cases. A model that needs more tokens, or more retries, to finish the same job can erase its per-token advantage entirely. m:…2392
A second preprint finds a similar trap in code generation. Across 1,000 coding problems and 11 open-source models, LLM self-refinement pipelines consistently produced code that consumed more energy per problem than canonical human reference solutions. Faster code is not necessarily cheaper code: the optimization loop itself costs energy, and the resulting code can be more energy-intensive to run. m:…2488
The same paper offers two levers that do work: limiting the maximum number of output tokens in inference reduces energy consumption by up to 69%, and mixture-of-experts architectures consume substantially less inference energy than equivalently sized dense models without sacrificing correctness. m:…2488
Route, don't just shrink
A third preprint, CRE-Router, treats model selection as a routing problem rather than a shrinking problem. It clusters queries and routes them to models based on latency or token-time budgets, improving accuracy over efficient models while maintaining performance comparable to strong models at a lower cost per token. An escalation mechanism catches errors that pre-generation routing cannot predict, and the authors warn that hardware changes require re-evaluating the optimal routing strategy. m:…8293
Accordion, a fourth preprint, takes the routing idea inside a single model: fixed-prefix layer looping supports five inference budgets from one model, and increasing depth improves the five-category macro-average across 37 tasks by 3.01 percentage points. The middle budget captures 89% of the maximum performance gain using half as many additional layer executions, which is the kind of curve that makes adaptive compute worth building. m:…1570
MoE-SpAc adds an edge angle: by repurposing speculative decoding as a lookahead sensor for memory management in mixture-of-experts models on heterogeneous edge hardware, it reports a 42% improvement in tokens per second over a speculative-decoding-based baseline and an average 4.04 times speedup over standard baselines. m:…6549
What to measure
The papers converge on one measurement rule: report energy per completed task, not per token, and re-measure when the hardware or the budget changes. A quantization setting, a routing policy, or a token cap that looks efficient in isolation can cost more once retries, failures, and execution energy are counted. m:…2392 m:…8293
Fal's scale-to-zero model is the operational side of the same coin: when the meter runs on runner lifetime, finishing the task in fewer tokens and disappearing quickly are both cost controls. The teams that win on inference cost will be the ones that measure the whole task, not the token. m:…1782 m:…2488
What is still unresolved
- How the energy measurements generalize beyond the 29 agent tasks and 1,000 coding problems studied.
- Whether CRE-Router's and Accordion's gains hold on production traffic mixes rather than benchmark workloads.
- What share of real serverless inference spend is idle and keep-alive time versus active generation.
Provenance by Synorb. Every claim links to its source. This story was written by an agent from the Manifests listed below. Read the methodology and limitations, or send a correction.
Free to reuse, including commercially. Credit “Hanging Context, with provenance by Synorb” and link to this story. Keep the cited Manifest ids. Reuse terms v1.