Diffusion-model compute patterns, time-aware quantization, TensorRT tuning, and FP8 kernels all allocate precision where it pays

Diffusion models spend unevenly, and that is useful

A preprint on diffusion language models reports that these models spend more computation on novel, rare, or incorrect outputs, and that the number of forward passes per token depends heavily on the task. Diffusion language models can write several tokens in one forward pass, and text copied or lightly edited from the input is cheaper to generate than open-ended writing. Generating a token for story writing in DiffusionGemma requires more than three times as many forward passes as for grammar correction, and highly tokenized languages require more passes per token. The novelty of output relative to input and word rarity explain much of the cost gap, and final answers are cheaper when they repeat the model's reasoning trace. m:…1646

The practical trick the preprint offers is to treat decoding speed as a signal. Wrong answers on math problems take more passes per token than correct ones, and weighting candidate answers by decoding speed beats majority voting by up to 4.2 points. That turns a cost observation into a quality method: the model's hesitation, measured in forward passes, carries information about whether the answer is right. A serving team can use that signal without retraining anything, which is the cheapest kind of improvement. m:…1646

Quantization that knows what time it is

A second preprint attacks the same problem from the precision side. Diffusion language models introduce unique quantization challenges due to their iterative denoising process, and quantization sensitivity in these models is both task- and time-dependent. Different downstream tasks exhibit distinct precision requirements across denoising steps. The authors propose T2Quant, a task and time-aware mixed-precision quantization framework that uses a label-free sensitivity signal to allocate activation precision across denoising steps under a fixed bit-width budget. Quantization sensitivity patterns can be predicted without ground-truth labels, and T2Quant predicts precision sensitivity using only a few unlabeled calibration prompts. m:…7352

The preprint says T2Quant improves the average accuracy of existing quantization methods by up to 6.7 percentage points, evaluated across 3 diffusion language models and 6 benchmarks. The important detail is the fixed bit-width budget: T2Quant does not buy accuracy with more bits, it reallocates the bits it already has to the steps and tasks that need them. That is the same move as the decoding-speed paper, one level down the stack. Both papers say the uniform setting is the wasteful setting. m:…7352

Compilers and kernels catch up

NVIDIA's TensorRT posts describe the production version of the same idea. Deploying a trained neural network involves more than translating each operator into a GPU kernel, the company says, and TensorRT engine building is a compilation problem with many legitimate answers. Engine building involves interdependent decisions about graph-IR transformations, fusion, implementation selection, memory placement, and quantization, and the process of making these interdependent decisions is called scheduling. The point is that there is no single correct compiled engine; there is a search space, and the search is the work. m:…4279

The Global Performance Tuner, released in NVIDIA TensorRT 11.1, automates that search. The company says the tuner provides automatic performance tuning, is accuracy-aware, performs end-to-end tuning, and customizes optimizations for different models. Accuracy-aware is the key qualifier: a tuner that only chases speed will happily trade away the quality the model was deployed for. The tuner's job is to find the fast configuration that keeps the answers right, which is exactly the trade the preprints are formalizing. m:…8508

AMD's ROCm post shows the same discipline at kernel level. The author details porting FP16 GEMM kernels to FP8 on AMD RDNA 4, noting that the FP8 WMMA instruction on RDNA 4 has changed compared to previous implementations. Smaller element sizes in FP8 allow for larger tiles to fit in LDS, and for deeper K slices to fit in LDS. The author compares four tile configurations against hipBLASLt, using GEMM shapes derived from transformer workloads. Porting FP16 GEMM kernels to FP8 on AMD RDNA 4 GPUs is possible, the author concludes, but the work is in the tile shapes, not in the idea. Lower precision is not a switch you flip; it is a configuration you tune, step by step, workload by workload. m:…8907