Token prices fell and GPU rents rose. The inference margin is caught in the middle

The spread

Silicon Data defines the crunch spread as token revenue per GPU-hour minus GPU rent per hour, and the spread has been tightening since May. The break-even throughput required to cover GPU rent has risen by 22% to 75% for open-weight models since May 31, 2026, and by 66% to 139% for proprietary models over the same period. Break-even throughput is calculated as GPU rent divided by token price, multiplied by 1,000,000 and divided by 3,600, so every move in either price shows up in the hurdle. m:…3964

The rent side is doing most of the damage. B300 rental costs increased by 51.9% between May 31 and October 22, 2026, and B300 rent explains approximately three-quarters of the higher break-even hurdle for open-weight models. H100 rental costs increased by 5.5% over the same period, a smaller move, but the direction is the same: the hardware costs more to rent while the output it produces sells for less. m:…3964

The token side is falling. The proprietary token index fell by 36.3% between May 31 and October 22, 2026, and the open-weight token index fell by 13.4% over the same period. Usage of open-weight models increased from approximately 16% of token volume in December 2025 to 74% in October 2026, which means the market is voting for the cheaper models even as the rent on the hardware that runs them climbs. m:…3964

Cheaper precision, same rent

An escape route is quantization, and OpenRouter's test suggests it is nearly free on quality. OpenRouter tested 1,455 provider endpoints across 470 models on October 20, 2026, and found no accuracy difference between fp8, fp4, and fp16 on 240 scored tasks for Qwen3.8 27B. All six endpoints tested for Qwen3.8 27B scored between 98.3% and 99.2% accuracy. LLM quantization stores model weights in fewer bits to reduce memory and hardware requirements, and the test says the quality cost of doing so is, for this model, undetectable. m:…5515

The price benefit is real but uneven. 8-bit was cheaper than 16-bit for 14 of 24 models served at both precisions, and 4-bit was cheaper than 8-bit for 26 of 33 models. But the provider affects the price more than the precision does. Users can pin precision using provider.quantizations to ensure consistent performance, and adding :floor to a model slug routes to the cheapest endpoint that passes the precision filter. m:…5515

Only 54 of 163 models with a labeled endpoint have a 16-bit option, and 610 of 865 unknown endpoints belong to closed-weight models. Median 24-hour uptime was 99.8% or higher at every labeled precision except fp32. The picture is a market where the cheap options work fine, but finding them takes routing, and the routing itself has become a product. m:…5515

The other squeeze: rate limits

Price is not the only constraint. A 429 Too Many Requests error indicates that a user has exceeded their request or token limit within the current time window, and default settings in the OpenAI Python SDK often fail to resolve 429 errors, with 23 of 60 requests failing in a test batch. Implementing a retry mechanism that waits for Retry-After or X-RateLimit-Reset headers ensures all requests in a batch complete successfully, and using model fallbacks allows a batch of 60 requests to complete in 13 seconds without waiting for rate limit resets. m:…5514

OpenRouter applies a single per-minute rate limiter to all free models on an account, and the daily free-model request limit is 50 for accounts without credit purchases and 1,000 for accounts with at least 10 credits. OpenAI counts max_tokens toward the tokens-per-minute limit, meaning generous values consume limits faster, and OpenAI and Anthropic return 429 errors when monthly spend caps are reached, which cannot be resolved by retrying. Disabling fallbacks or narrowing provider lists can reduce the capacity available to absorb traffic bursts. m:…5514

The demand side does not blink

None of this is slowing demand. 51% of U.S. AI-native leaders run large-scale inference as a production workload, and 94% of U.S. AI-native leaders expect their compute needs to increase in the next six to 12 months. More than a third, 37%, expect compute demand growth of 10x or more. 96% of U.S. AI-native leaders use more than one provider or platform, which is the multi-provider reality that makes routing and fallback strategies worth building. m:…7081

The overhead is the tax on that demand. 60% of AI-native leaders spend more than a quarter of their engineering time on infrastructure work, and 34% report it takes four days or more to move a trained model into a live endpoint. Nscale will become Figure's preferred compute provider for its Helix models and humanoid robots, and Nscale has entered into an agreement to acquire Anyscale. The inference business is consolidating around whoever can make the plumbing disappear, because the rent is not going down. m:…7081