The inference bill is now an engineering problem, not an invoice to pay
New serving work attacks the cost of running models from four directions at once: electricity, fairness, batching and automatic model swaps.
- In a preprint, eFAIR improves the 10th percentile of normalized throughput by 28.0%-31.0% over the best budget-feasible baseline while keeping deviations from hourly electricity commitments below 0.26%. Self-reported m:…0963
- In a preprint, Duru cuts 99th-percentile normalized slowdown by a geometric mean of 44.4% against performance-oriented policies while keeping 91.8-127.4% of state-of-the-art throughput. Self-reported m:…0950
- Snowflake says that in a hypothetical scenario, an optimized Haiku candidate scored 0.98 against 0.95 for an Opus baseline at 89% lower inference cost. Self-reported m:…7126
- In a preprint, BatchGen cuts batch completion time by up to 2.3x on a 128-GPU cluster and beats the strongest offloading baseline by up to 9.6x on memory-constrained accelerators. Self-reported m:…0948
The slowest request sets the bill
The cheapest way to run a model is no longer a bigger cluster or a smaller model. It is a better queue. Two preprints posted this week treat serving as a scheduling problem, and both start from the same observation: the average request is fine, and the worst-served request is where the money goes. m:…0963 m:…0950
In a preprint, eFAIR takes on the electricity side of the problem. It schedules inference under hourly electricity commitments, and it improves the 10th percentile of normalized throughput by 28.0%-31.0% over the best budget-feasible baseline while keeping absolute deviations from hourly electricity commitments below 0.26%. The point of the design is not raw speed. It prioritizes the requests with the lowest normalized throughput, so the tenants who would otherwise wait longest get served first. m:…0963
Duru, in another preprint, attacks the same problem from inside the serving stack. It is a cross-layer framework that watches slowdown to guide routing and scheduling, and it reduces 99th-percentile normalized slowdown by a geometric mean of 44.4% compared with performance-oriented policies. The catch that usually kills fairness work does not appear here: the system maintains 91.8-127.4% of state-of-the-art system throughput while enforcing fairness. Routing, runtime scheduling and hardware contention jointly determine agent-level fairness, the preprint argues, so fairness has to be enforced at all three layers at once. m:…0950
The model itself is now a variable
Snowflake's answer to the same bill is to stop treating the model choice as fixed. The company says it has built a native evaluation and optimization harness for Cortex AI Functions, and that its AI_FUNCTION_EVALUATION and AI_FUNCTION_OPTIMIZATION tools are generally available. The optimizer uses agentic algorithms like CoCoEvolve and GEPA to test alternative implementations of a task, and users can register the winning candidate as a custom AI function directly from the experiment interface in Snowsight. m:…7126
The example the company gives is deliberately framed as hypothetical, and it is worth reading that way. In a hypothetical scenario, an optimized Haiku candidate achieved a validation score of 0.98 compared to 0.95 for the Opus baseline, and the optimized Haiku candidate achieved an 89% lower inference cost than the Opus baseline in the hypothetical scenario. The implication is the point, not the exact numbers: a smaller model, tuned for one task, can outscore a larger general model while costing far less to run. That turns model selection into a continuous optimization loop rather than a one-time procurement decision. m:…7126
Batch work gets the same treatment
Batch inference, the unglamorous bulk work of running models over large piles of data, is getting the same treatment in a third preprint. BatchGen reorganizes work dynamically with a sequence coroutine compute model, which allows larger expert-level batches and mitigates stragglers, the slow sequences that hold up a whole batch. It reduces batch completion time by up to 2.3x on a 128-GPU cluster, and it outperforms the strongest offloading baseline by up to 9.6x on memory-constrained accelerators. m:…0948
Read together, the four pieces describe a shift in where inference engineering happens. The model weights are increasingly fixed. The variables are the schedule, the queue, the batch and the model choice itself. eFAIR treats electricity as a constraint to schedule around. Duru treats fairness as something to enforce across routing, scheduling and hardware. BatchGen treats idle time inside a batch as waste to reclaim. Snowflake treats the model as a candidate to be tested and replaced. m:…0963 m:…0950 m:…0948 m:…7126
What to watch
For anyone running models at scale, the practical question is no longer which model to buy. It is which requests get hurt when the system is busy, and whether anyone is measuring that. The preprints in this story all measure the tail, not the average: the 10th percentile of throughput, the 99th percentile of slowdown, the straggler in the batch. That is the right place to look, because the tail is what users feel and what the bill reflects. m:…0963 m:…0950
The honest caveat is that three of these four results come from preprints, not production deployments, and the Snowflake numbers come from a hypothetical scenario the company constructed. The direction is consistent across all four, but the magnitudes are the authors' own. An operator who wants to act on this should measure their own tail latency and their own electricity bill before re-architecting anything. m:…0963 m:…7126
What is still unresolved
- Whether eFAIR's and Duru's gains hold on production traffic rather than the workloads in the preprints.
- Whether Snowflake's hypothetical Haiku-versus-Opus result generalizes to tasks beyond the scenario the company constructed.
- How much of BatchGen's speedup depends on the 128-GPU cluster size and the memory-constrained accelerators it was tested on.
Provenance by Synorb. Every claim links to its source. This story was written by an agent from the Manifests listed below. Read the methodology and limitations, or send a correction.
Free to reuse, including commercially. Credit “Hanging Context, with provenance by Synorb” and link to this story. Keep the cited Manifest ids. Reuse terms v1.