Inference splits into specialists as demand outruns commodity tokens

Tokens are a commodity, everything else is not

Erik Bernhardsson, CEO of Modal, told the 20VC podcast that the demand for AI inference is effectively infinite, and that selling raw inference tokens is a commodity business with no switching costs. The two claims together explain the strategic bind: demand keeps growing, but the simplest product built to serve it cannot hold a margin, because a customer can move to the next cheapest token seller without friction. m:…7605

Bernhardsson's answer is fragmentation. He says the inference market will fragment into specialized sub-markets for low latency, batch processing, and fine-tuning, and that Modal differentiates by offering tools beyond tokens, including sandboxes, post-training, and reinforcement learning. Modal aggregates demand across thousands of customers to manage GPU capacity efficiently, which is presented as the operational answer to a capacity problem that individual customers cannot solve alone. m:…7605

The financial picture he describes is mixed in a revealing way. Modal is profitable on a profit-and-loss basis but has negative cash flow due to GPU capacity investments, and its sandbox product reached a 300 million dollar run rate by mid-2025, according to Bernhardsson. He also argues that the 90% software margins seen in Silicon Valley a decade ago are effectively gone due to the layered nature of the AI stack, and that CUDA is not a true moat for NVIDIA, though the broader ecosystem of kernel optimizations and framework support is. Hardware prices for GPUs will eventually decrease in the long run, despite current supply shortages, he says. m:…7605

Latency as the product

FriendliAI's integration with Vercel AI Gateway shows what a latency-specialized pitch looks like in numbers. FriendliAI is now available on Vercel AI Gateway, and the company says it recorded a median time to first token of 266 ms for GLM-5.3 on the gateway. That median was the lowest among 22 providers on Vercel AI Gateway, and the company says FriendliAI was 19.8 times faster than the average of other providers over the past week. m:…4567

The integration is also about control, not just speed. Users can pay for inference using Gateway credits or their own FriendliAI API key, the order configuration in providerOptions allows for provider fallbacks, and the only configuration restricts requests to a specific provider. If a request using a FriendliAI bring-your-own-key fails, Gateway retries with system credentials. For a founder, that combination of routing, fallback, and billing flexibility is the product: latency is the headline, but the ability to steer traffic when a provider stumbles is what keeps an application alive. m:…4567

Note the scope of that speed claim: a median for one model, GLM-5.3, on one gateway, over one week, as reported by the provider itself. It is a signal for interactive applications, not a general claim of fastest for every workload. m:…4567

Real time needs a different shape

fal's real-time GPU inference offering shows a second specialization: interactivity itself. Realtime endpoints bypass the queue and keep the runner warm, and the @fal.realtime decorator allows clients to send back-to-back inputs over one WebSocket. WebRTC is used for live video and audio streaming while the WebSocket handles signaling, and realtime endpoints use binary msgpack by default for serialization. Queue features such as retries and webhooks do not apply to realtime endpoints, which is the trade-off for skipping the queue. m:…2317

Keeping runners warm is a capacity decision with a price. A min_concurrency setting keeps a fixed number of runners alive at all times, and a concurrency buffer keeps spare runners warm beyond current demand, while the first connection to a realtime endpoint may still experience a cold start. Browser clients should use proxy URLs or short-lived tokens instead of API keys to ensure security, according to fal, and deploying custom realtime endpoints requires approval from the fal team. The picture is of infrastructure tuned for streams that cannot wait, where idle capacity is a feature rather than waste. m:…2317

This is the low-latency sub-market Bernhardsson described, made concrete. A batch job can tolerate a queue; a live voice or video interaction cannot, so the provider charges in effect for readiness: warm runners, persistent connections, and binary serialization. Founders choosing between providers for a real-time product are therefore not buying tokens at all; they are buying a connection shape and a warmth guarantee. m:…7605 m:…2317

Down to the silicon

A preprint on chiplet architecture pushes specialization one layer deeper, to the hardware that serves long contexts. GPU compute-rich architectures are fundamentally mismatched with the memory-bound nature of decode-phase attention, according to the AMMA authors, because decoding spends its time moving data rather than crunching numbers. Replacing GPU compute dies with HBM-PNM cubes doubles available memory bandwidth for attention workloads, on their account. m:…1284

The reported results are large: the AMMA architecture achieves 15.5 times lower attention latency compared to NVIDIA H100, and 6.9 times lower energy consumption compared to NVIDIA H100. Those figures describe attention serving at 1 million context lengths, a workload where memory bandwidth dominates. Whether chiplet designs like this reach production at scale is unproven, but the direction fits the fragmentation thesis: when one workload's bottleneck is memory bandwidth and another's is raw compute, a single general architecture serves both badly. m:…1284

For founders, the practical takeaway is to buy against a workload, not a market average. A token price tells you little about first-token latency, warm-runner guarantees for a live stream, or attention latency at very long context. The providers in this set are each optimizing a different slice, and the market Bernhardsson describes is one where the slice, not the token, is the unit of competition. m:…7605 m:…4567 m:…2317 m:…1284