Stop ranking inference hardware by FLOP/s. It is the wrong number

Reproducibility at near-stock speed

Detent, a preprint on LLM serving, addresses a problem that quietly corrupts RL training and model evaluation: serving engines like vLLM and SGLang suffer from batch-dependent output variance, so the same prompt can produce different outputs depending on what else is in the batch. Detent achieves batch-invariance by programmatically fixing kernel selection values, and its latency is 1.04 to 1.17 times that of standard engines, compared with 1.67 to 10.3 times for existing deterministic modes. In benchmarks on Llama-3.1-70B and Qwen3-30B-A3B, the overhead is small enough that reproducibility stops being a luxury. m:…4180

The payoff is concrete. Detent enables bit-for-bit reproducible RL rollouts for models up to Llama-3.1-70B, and BF16 GRPO training remains stable on Detent where standard deterministic modes cause collapse. For teams doing RL training or rigorous model evaluation, output consistency is non-negotiable, and until now the price of that consistency was either a large latency penalty or a training run that could not be reproduced. Detent suggests the tradeoff was an engineering artifact, not a law of nature, and that fixing kernel selection is enough to remove most of it. m:…4180

FLOP/s understates the new hardware

Epoch AI Research has launched the Inference Performance Index, a framework for evaluating inference capacity that prioritizes memory and communication throughput over raw compute speed. The Nvidia GB300 provides approximately 25 times the inference capacity of the Nvidia H100 per chip on modeled agentic workloads, Epoch says, while having only 2.5 times the 8-bit FLOP/s. Comparing FLOP/s alone underestimates the GB300's inference capacity relative to the H100 by a factor of ten. Memory and inter-chip communication are often more critical than arithmetic throughput for inference performance, the authors argue, and inference improvements can be larger than changes in raw compute speed suggest. m:…6790

The index focuses on agentic workloads and expresses scores in GB300 equivalents, modeling prefill and decode phases across representative architectures. The Nvidia GB300 NVL72 has approximately 1.9 times the modeled inference capacity per GPU of the Nvidia B300 HGX, and the difference is partly architectural: the GB300 NVL72 connects groups of 72 GPUs through high-bandwidth links, while the B300 HGX connects groups of eight. Epoch also reports that agentic API keys consumed roughly seven times as many tokens as human API keys in late September 2026, which is why the index targets agentic workloads in the first place. The agents are now the dominant consumers of inference. m:…6790

Quantization: the metric can lie

Applied Compute's work on quantization-aware distillation carries a warning that rhymes with the retrieval story: the metric can improve while the thing it measures does not. Quantization-aware distillation can reduce the held-out KL of NVFP4 models against their BF16 base by upwards of 40%, the team reports, but large KL reductions in quantized models do not always correspond to recovered task accuracy. Engine mismatch between training and rollout engines can introduce significant quantization error, caused by differences in kernel selection and rounding. Their proposed fix is deployment-informed distillation, which uses the rollout engine's probabilities to calculate gradients, helping the model adapt to deployment-specific quantization errors. m:…0753

The same report shows how granular the work has become. Mixed-precision search on Qwen 3.8 27B allowed components to use NVFP4, FP8 or BF16, and keeping attention components in BF16 reduced KL against the original model by 49 to 54% across four quantization recipes. Quantization reduces memory bandwidth pressure by decreasing the amount of data moved around the chip, and increases arithmetic throughput through specialized hardware acceleration. The lesson is that quantization is no longer a single switch. It is a per-component decision, and the right answer depends on the engine that will actually run the model. m:…0753

One replica model, compressed caches

Polyaxon is developing a shared replica-group model to standardize workload configuration across inference, training and reinforcement learning on Kubernetes, with each named replica group having its own instance count and workload configuration. The first target is disaggregated inference using SGLang and Model Gateway, and the example deployment uses 4 prefill replicas and 8 decode replicas, requiring 12 GPUs across 13 pods. Polyaxon plans to integrate with Volcano for gang placement, KAI Scheduler for GPU scheduling and topology constraints, and Kueue for quota admission. The idea is that inference, training and RL stop being three different deployment shapes and become one model with different replica groups. m:…2647

LowRAM, a preprint on KV cache compression, shows how much headroom remains in the serving layer itself. It compresses KV caches by predicting values from keys within each attention head and applying cross-layer low-rank key compression, achieving up to 5 times higher peak decode throughput at 4 times KV compression, and composing with quantization and eviction to reach 26.9 times compression at 94.5% of baseline performance on long contexts. Taken together, the five sources describe an inference stack being rebuilt from the bottom up: reproducible serving, honest capacity measurement, deployment-aware quantization, unified orchestration and compressed caches. The teams that adopt these together will run the same models materially cheaper than teams that adopt any one alone. m:…5407