Inference is not limited by FLOPs. It is limited by memory bandwidth.

The math of decode

Featherless AI's analysis of inference economics starts with a correction: "Almost every inference deployment is sized by FLOPs, and almost none of them are limited by FLOPs." The decode step is bandwidth-bound because the accelerator must read every parameter the token activates. For a 320 billion parameter mixture-of-experts model, producing one token takes 36 GFLOPs of arithmetic and 18 GB of weight movement. Compute capacity far exceeds memory throughput, so the memory system, not the arithmetic units, sets the speed limit. As the post puts it: "Decode is bandwidth-bound, and not marginally." m:…4727

The second constraint is the KV cache, which scales linearly with sequence length and often becomes the binding limit on how many conversations a node can run at once. Featherless puts the KV cache for a 256K context window at approximately 23.6 GB per sequence, so a node with 1024 GB of memory can support 27 concurrent conversations at full 256K context. Batching amortizes the cost of reading model weights but does not reduce the per-sequence KV cache cost, and decode only becomes compute-bound at batch sizes in the low hundreds. Increasing batch size to chase throughput also increases the time to last token, which is the latency users actually feel. m:…4727

The economics follow from utilization. A dedicated node pays for itself at 759 million output tokens per month at a $10 per million token price point, and dedicated inference nodes are more cost-effective for steady demand, long contexts, or models not available through per-token APIs. Featherless's summary is blunt: "Utilisation is the whole story." A node that sits idle is expensive no matter how fast its memory is. m:…4727

The configuration problem

Knowing the bottleneck does not solve the serving problem, because serving a model means choosing an engine, a parallel layout, a precision, and scheduler settings, with thousands of potential combinations. AMD's ROCm team says measuring every candidate in its searches would take about 25,000 GPU-hours, with each trial tying up one or more multi-GPU nodes for minutes. As AMD's Emad Barsoum put it: "A guess that misses costs you extra GPUs or a missed latency target." Most teams currently try a few guesses and keep the best, which is why AMD built InferaSim to automate the search for the best configuration without the cost of exhaustive testing. m:…6834

The same week, NVIDIA's developer blog addressed the latency problem from the model side, for image generation. Generating an image with a diffusion model requires multiple sequential passes through a model with billions of parameters, and higher resolutions increase attention costs. The post walks through optimizing FLUX 3 Image Klein, Black Forest Labs' image editing and generation model, on Blackwell GPUs. The pattern is the same as in text inference: the model is not the bottleneck, the passes through it are. m:…5420

The deployment layer

Runpod's launch of the Runpod Hub completes the picture at the deployment layer. The Hub is a marketplace for open-source AI repositories that deploys directly from a GitHub repo onto Runpod's serverless infrastructure, automating containerization and provisioning. Initial prebuilt repositories include ComfyUI for image generation, OpenAI's Whisper, the Mochi 1 video generation model, Axolotl for fine-tuning, and Ollama. Community members can publish public repositories to the Hub. m:…4324

Put together, the week's inference news is not about a new model. It is about the plumbing around the model: memory bandwidth as the real constraint, configuration search as the hidden cost, and one-click deployment as the way open-source models reach users. Teams that size their deployments by FLOPs will keep buying the wrong thing. m:…4727 m:…6834 m:…4324