Inference Is a Commodity Now. The Contest Is Serving Cost.

The market now behaves like a commodity

Mistral CEO Arthur Mensch told the 20VC podcast that the inference market behaves very much like a commodity market. That behaviour is exactly what draws a cost contest: when buyers can switch between near-identical serving, serving cheapest is what wins. Mensch also argues the constraint is not energy. He puts energy at around 10% of the total cost of owning compute, and says the cost of energy is not the one thing that matters in the economics of a project: there is not enough compute, and the bottleneck is compute and capital. The fight, in other words, is over how much useful inference each unit of compute produces. m:…2005

The same pressure is visible in the model supply. Mensch says the open-weight frontier is catching up with the closed frontier, and in many domains is actually above it. Mistral says around 80% of its token traffic comes from its own models. When capable open-weight models are widely available, differentiation moves downstream from the weights themselves to serving: who can run them cheapest, fastest and most reliably at scale. m:…2005

Caching is the first lever

FriendliAI's account of KV caching shows where the first savings come from. KV caching stores the attention state of an agent's growing history so each call does not recompute it, and prefix caching reuses that retained state when a later request begins with an identical token sequence under the same model and compatible serving configuration. The company says reusing cached history lowered input token costs by 80% in its 1,000-task example. For agents, whose histories grow with every tool call, the saving compounds: the longer the conversation, the more recomputation caching avoids. m:…8665

The rest of FriendliAI's toolkit is about stretching the cache further. Hierarchical caching extends capacity by retaining KV state in host DRAM and disk, beyond GPU memory. Cache-aware routing sends each request to the worker that already holds the matching history, which can respond faster even with a longer queue. Sparse attention, such as the DeepSeek Sparse Attention used in GLM-5, selectively reads from the main KV cache to reduce memory traffic, and multi-head latent attention stores a compressed representation of attention state. KV cache quantisation can reduce memory further by using fewer bits per stored value, where supported. The scale of the problem is visible in one figure: for GLM-5, a single history needs approximately 2.74 GiB of main-attention KV state at 32K tokens and 10.97 GiB at 128K tokens, assuming BF16 storage. m:…8665

Compression and faster kernels

A preprint on AutoCompress takes the same cost question to the model itself. AutoCompress is a hardware-guided compression framework that uses differentiable architecture search and serving-engine cost models, building latency, throughput and memory metrics directly into the search instead of compressing first and measuring later. Compared with the Minitron pruning framework, it improves average accuracy across eight benchmarks by 6.57 percentage points on RTX 5070, 2.35 on RTX 5080 and 0.57 on H100, while using 2 to 3.3 times fewer architecture-search tokens than the best available baseline. m:…2288

The headline result is a large model made servable on one card. AutoCompress compressed Nemotron 3.5 Super from 120B to 74B parameters for single-B200 BF16 serving at 128K context, retaining 95% of its mean benchmark score. Two other preprints attack the serving layer from different angles. Uni-LaSpec, a load-adaptive speculative decoding framework for vision-language models, improves throughput by up to 5.11x for text and 3.24x for multimodal tasks compared with autoregressive decoding, and under bursty arrivals it reduces mean end-to-end latency by 47.6% relative to linear verification. Selecting eight visual tokens retains at least 98.6% of the acceptance rate compared with using the full image. Transformer-FSQ, which quantises the KV cache into discrete integer codes, achieves 12x to 97.5x KV-cache compression and outperforms five baseline families on RULER long-context retrieval, with a fused Triton kernel reaching 1.01x to 1.54x the throughput of native SDPA attention. m:…2288 m:…0837 m:…5100

Caching can also collapse

A fourth preprint is a warning about the caching lever itself. It describes prefix-cache collapse: bursty traffic can push an inference engine into a persistent all-miss state, where the cache stops helping at all and the backlog becomes unrecoverable. In the trapped state the gap between turns grows linearly, and the engine's throughput overstates the actual recovery threshold, so the dashboard can look healthier than the system is. m:…1968

The fix it identifies is operational, not architectural: a session cap is sufficient to prevent prefix-cache collapse and restore reuse without shedding arrivals. That is the shape of the new contest. Inference is cheap enough to be a commodity, expensive enough that caching, compression and kernels decide who profits, and fragile enough that a burst of traffic can take the savings away unless the operator caps concurrency before the trap sets in. m:…1968