The agentic serving bill: measure the finished job, not the token

The per-token trap

The cheapest token is not the same as the cheapest finished job. In a preprint, researchers studying agentic serving report that agentic requests use 13 times the energy of non-agentic requests, a gap large enough to swamp most per-token savings. The same line of work offers a four-parameter analytical model, called SIMPLE, that predicts inference energy across 13 dense models on H100 graphics processors with an R2 of 0.969 and a mean absolute percentage error of 11.97 percent, without calibrating the model separately for each model. That combination matters because it gives operators a way to estimate the bill before they commit to a serving pattern, rather than discovering it after the workload has scaled. m:…5274

The model also breaks energy into parts that point at different fixes. Its authors report that grouping key-value cache entries cuts per-token energy by up to three times compared with standard multi-head attention, which suggests that architecture choices made long before serving starts still shape the bill. The implication is plain: when a team compares two serving setups, the comparison should start from the workload the team actually runs, including how many turns an agent takes and how much context it carries, not from a single-token benchmark that strips those choices away. m:…5274

Routing for completed work, not tokens

Routing is a second place where the per-token view misses the cost. GoodServe replaces greedy routing with a globally coordinated, goodput-oriented policy for agentic language-model applications, where utility turns on whether a request meets its end-to-end deadline. The system uses a predict-and-rectify mechanism that accounts for output-length uncertainty and prefix-cache reuse, cutting wasted compute. On heterogeneous GPU testbeds it reaches up to 31 percent higher goodput than existing greedy routers. Moving a request at risk by token identifier costs 7 to 15 times less than transferring its key-value cache, and routing overhead stays at 5 milliseconds per request at 10,000 requests per second across 512 simulated instances. The lesson is that placement across a pool matters as much as the model: work that misses its deadline, however cheap per token, does not count. m:…9578

Branch parallelism shows the same measurement problem inside a single request. PACE treats parallel branches as opportunistic width, admitted per step only when slack budgets allow, instead of admitting them eagerly. On Qwen3-32B it improves goodput by 1.77 times over serving without branch parallelism, and by 1.48 times over eager admission while maintaining 99 percent service-level-objective attainment. Branch-level scheduling decouples compute from memory, so width can change dynamically without memory reclamation. A branch admitted without regard for co-batched requests can degrade the work around it, so the useful comparison is completed work under deadline, not raw parallelism. m:…0596

Metering shows why a token count cannot carry the comparison. Under batched serving, no per-request meter can be simultaneously additive, independent of co-resident traffic, and exactly cost-reflective. The proposed architecture uses reference inference units and occupancy-indexed ratings: occupancy-indexed rating limits statement energy error to 9 percent under measured traffic, compared with 70.5 percent under fixed-occupancy rating, and tenant mispricing falls from 7.36 times under flat token counting to 1.06 times after occupancy-based reconciliation. A provider that bills only tokens will therefore misprice tenants whenever sharing changes the cost of a request, which is exactly the condition agentic serving creates. m:…1487

What to measure instead

The same logic applies to reasoning budgets. In the SIMPLE preprint, the authors report that intelligence per joule peaks at an intermediate reasoning budget and then falls by up to 3.14 times across a fourfold range of reasoning budgets. In other words, giving an agent more room to think improves accuracy for a while, then costs energy faster than it buys accuracy. This suggests a practical rule for builders: set the reasoning budget where the accuracy gain still justifies the energy, and treat a larger budget as a purchase with a diminishing return, not as a free improvement. m:…5274

Put together, these studies point to one measurement discipline. Report energy per request and goodput under deadlines for the actual task mix, under actual sharing, and with metering that accounts for occupancy. A model that is cheaper per token but misses deadlines, wastes routing, or is mispriced under sharing is not cheaper. A reasoning budget that helps at one size can hurt at a larger one. None of this requires new hardware to act on: it requires measuring the unit the user actually pays for, which is the finished job. m:…5274 m:…9578 m:…1487