Instead of buying more GPUs, vendors are selling ways to use the ones you have better

Sharing the GPU

The NVIDIA CUDA 13.4 release is a small announcement with a large implication for platform teams. NVIDIA says it released CUDA 13.4 with memory-enforced GPU sharing for Kubernetes. Assigning an entire GPU to a single workload can result in idle capacity, according to NVIDIA, and many AI workloads require only a fraction of a GPU. GPU sharing allows platform teams to serve more workloads with existing infrastructure, and multi-instance GPU splits a GPU into hardware-isolated instances that are transparent to the application. The word memory-enforced matters here: sharing without enforcement lets one workload crowd out the others, which is why teams have historically been reluctant to share. A sharing mechanism that enforces memory limits turns a risky optimisation into a platform default. m:…8143

NVIDIA says its GPUs support multiple approaches to sharing, which is a reminder that there is no single sharing mode to choose. The practical question for a platform team is which workloads can share safely and which need isolation, and that question has to be answered with measurements rather than defaults. A second NVIDIA post shows where those measurements lead for a specific model. Decoding performance for Ling-3.0-flash on NVIDIA DGX Spark is memory-bound, reading model weights is the primary driver of execution time during inference, and MoE computation and output-layer projection are the two operations that most significantly impact inference latency. The evaluation was conducted at batch-size-1. That detail matters: a system that is memory-bound at batch-size-1 will not be fixed by adding compute, and a team that buys more GPUs for a memory-bound workload is paying for the wrong resource. m:…8143 m:…3956

Faster decoding, smaller drafts

AMD takes the same utilisation logic into speculative decoding. The company says speculative decoding performance is limited by the quality of the draft model, AMD supports EAGLE3 speculative decoding draft model training on AMD Instinct MI350X GPUs, and AMD Quark FP8 quantization is used for draft model optimization. The chain of reasoning is short: a better draft model means more tokens accepted per step, which means fewer full-model passes for the same output. Quantization matters in the same chain, because a smaller draft model is cheaper to run and easier to keep in memory. For a developer, the actionable part is not the brand of the draft model but the measurement: track acceptance rate alongside latency, because a draft model that looks fast in isolation can be slow in a full system if its suggestions are mostly rejected. m:…4823

A second AMD post goes one level deeper, into the kernels themselves. Keeping matrix units busy on compute-bound shapes depends on overlapping data preparation and computation, according to AMD, and tiled GEMM operations compute on one pair of operand tiles while preparing the next. Resident waves often reach synchronization points in phase, causing both to prepare data or wait simultaneously, and extra buffering does not by itself establish the execution timing needed to hide data preparation. That is a useful corrective to a common instinct: adding buffers feels like optimisation, but if the waves still synchronise, the hardware still waits. Warp-pipelining, as described in the post, is about timing the preparation so the compute units are rarely idle. The lesson generalises beyond AMD hardware: utilisation is a scheduling problem, not just a capacity problem. m:…9699

The rent versus buy math

fal pricing post puts a price on the same question from the other side. The company says renting an H100 costs $4.50 per GPU-hour at list price, with rates as low as $2.49, and purchasing an H100 card buys about 6,000 to 8,200 rented GPU-hours before accounting for server, power or staff costs. A new NVIDIA H100 80 GB PCIe card lists at about $27,000 to $37,000 from US retailers, according to fal, and the H200 with 141 GB is often a better buy for the largest models compared with the H100. The comparison is explicitly incomplete: it excludes server, power and staff costs, which are exactly the costs that make a purchase look cheaper than it is. A team that uses the 6,000 to 8,200 hour figure as a break-even point without adding those costs will buy too early. m:…3323

What to measure first

This suggests the first purchase decision for many teams is not a GPU at all, but a measurement. GPU sharing allows platform teams to serve more workloads with existing infrastructure, according to NVIDIA, but only if the team knows which workloads are memory-bound, which are compute-bound, and which can share without degrading each other. Reading model weights is the primary driver of execution time during inference for the Ling-3.0-flash case, and speculative decoding performance is limited by the quality of the draft model, according to AMD. Those two sentences are a measurement plan: profile weight movement, profile draft acceptance, and only then decide whether to buy, rent or share. m:…8143 m:…3956 m:…4823