---
title: Stop ranking inference hardware by FLOP/s. It is the wrong number
description: Detent makes serving bit-for-bit reproducible at near-stock speed, Epoch's new index says FLOP/s understates the GB300 by tenfold, and quantization work shows KL gains do not always mean accuracy gains.
canonical: https://hangingcontext.com/stories/2026-11-10/inference-reproducibility-serving/
last-updated: 2026-11-10
url: https://hangingcontext.com/stories/2026-11-10/inference-reproducibility-serving/
published: 2026-11-10T12:34:32Z
edition: 2026-11-10T12
newsroom: artificial-intelligence
beat: infrastructure
trust: confirmed
source_count: 5
cited_manifest_ids: ['1794311678756684180', '1794254702769366790', '1794251095672340753', '1794238119626235407', '1794311279343682647']
---

# Stop ranking inference hardware by FLOP/s. It is the wrong number

*Detent makes serving bit-for-bit reproducible at near-stock speed, Epoch's new index says FLOP/s understates the GB300 by tenfold, and quantization work shows KL gains do not always mean accuracy gains.*

Confirmed · 5 Manifests · Artificial Intelligence · Published 2026-11-10 12:34 UTC

## Key facts

- Detent runs at 1.04 to 1.17 times the latency of standard engines, against 1.67 to 10.3 times for existing deterministic modes, and enables bit-for-bit reproducible RL rollouts up to Llama-3.1-70B. (self-reported) [cites: 1794311678756684180]
- Epoch AI says the Nvidia GB300 has about 25 times the inference capacity of the H100 per chip on modeled agentic workloads, and FLOP/s alone understates it by a factor of ten. (self-reported) [cites: 1794254702769366790]
- Quantization-aware distillation can cut held-out KL of NVFP4 models against their BF16 base by over 40%, but large KL reductions do not always correspond to recovered task accuracy. (self-reported) [cites: 1794251095672340753]
- LowRAM reaches up to 5 times higher peak decode throughput at 4 times KV compression, and 26.9 times compression at 94.5% of baseline performance when composed with quantization and eviction. (self-reported) [cites: 1794238119626235407]

## Reproducibility at near-stock speed

Detent, a preprint on LLM serving, addresses a problem that quietly corrupts RL training and model evaluation: serving engines like vLLM and SGLang suffer from batch-dependent output variance, so the same prompt can produce different outputs depending on what else is in the batch. Detent achieves batch-invariance by programmatically fixing kernel selection values, and its latency is 1.04 to 1.17 times that of standard engines, compared with 1.67 to 10.3 times for existing deterministic modes. In benchmarks on Llama-3.1-70B and Qwen3-30B-A3B, the overhead is small enough that reproducibility stops being a luxury. [cites: 1794311678756684180]

The payoff is concrete. Detent enables bit-for-bit reproducible RL rollouts for models up to Llama-3.1-70B, and BF16 GRPO training remains stable on Detent where standard deterministic modes cause collapse. For teams doing RL training or rigorous model evaluation, output consistency is non-negotiable, and until now the price of that consistency was either a large latency penalty or a training run that could not be reproduced. Detent suggests the tradeoff was an engineering artifact, not a law of nature, and that fixing kernel selection is enough to remove most of it. [cites: 1794311678756684180]

## FLOP/s understates the new hardware

Epoch AI Research has launched the Inference Performance Index, a framework for evaluating inference capacity that prioritizes memory and communication throughput over raw compute speed. The Nvidia GB300 provides approximately 25 times the inference capacity of the Nvidia H100 per chip on modeled agentic workloads, Epoch says, while having only 2.5 times the 8-bit FLOP/s. Comparing FLOP/s alone underestimates the GB300's inference capacity relative to the H100 by a factor of ten. Memory and inter-chip communication are often more critical than arithmetic throughput for inference performance, the authors argue, and inference improvements can be larger than changes in raw compute speed suggest. [cites: 1794254702769366790]

> **From Synorb** — Ask what Synorb covers. The MCP tool synorb-stream-search answers questions like "do you have this publisher?" without using any Manifest allowance. [Connect and ask](https://synorb.com/connect/start)


The index focuses on agentic workloads and expresses scores in GB300 equivalents, modeling prefill and decode phases across representative architectures. The Nvidia GB300 NVL72 has approximately 1.9 times the modeled inference capacity per GPU of the Nvidia B300 HGX, and the difference is partly architectural: the GB300 NVL72 connects groups of 72 GPUs through high-bandwidth links, while the B300 HGX connects groups of eight. Epoch also reports that agentic API keys consumed roughly seven times as many tokens as human API keys in late September 2026, which is why the index targets agentic workloads in the first place. The agents are now the dominant consumers of inference. [cites: 1794254702769366790]

## Quantization: the metric can lie

Applied Compute's work on quantization-aware distillation carries a warning that rhymes with the retrieval story: the metric can improve while the thing it measures does not. Quantization-aware distillation can reduce the held-out KL of NVFP4 models against their BF16 base by upwards of 40%, the team reports, but large KL reductions in quantized models do not always correspond to recovered task accuracy. Engine mismatch between training and rollout engines can introduce significant quantization error, caused by differences in kernel selection and rounding. Their proposed fix is deployment-informed distillation, which uses the rollout engine's probabilities to calculate gradients, helping the model adapt to deployment-specific quantization errors. [cites: 1794251095672340753]

The same report shows how granular the work has become. Mixed-precision search on Qwen 3.8 27B allowed components to use NVFP4, FP8 or BF16, and keeping attention components in BF16 reduced KL against the original model by 49 to 54% across four quantization recipes. Quantization reduces memory bandwidth pressure by decreasing the amount of data moved around the chip, and increases arithmetic throughput through specialized hardware acceleration. The lesson is that quantization is no longer a single switch. It is a per-component decision, and the right answer depends on the engine that will actually run the model. [cites: 1794251095672340753]

## One replica model, compressed caches

Polyaxon is developing a shared replica-group model to standardize workload configuration across inference, training and reinforcement learning on Kubernetes, with each named replica group having its own instance count and workload configuration. The first target is disaggregated inference using SGLang and Model Gateway, and the example deployment uses 4 prefill replicas and 8 decode replicas, requiring 12 GPUs across 13 pods. Polyaxon plans to integrate with Volcano for gang placement, KAI Scheduler for GPU scheduling and topology constraints, and Kueue for quota admission. The idea is that inference, training and RL stop being three different deployment shapes and become one model with different replica groups. [cites: 1794311279343682647]

LowRAM, a preprint on KV cache compression, shows how much headroom remains in the serving layer itself. It compresses KV caches by predicting values from keys within each attention head and applying cross-layer low-rank key compression, achieving up to 5 times higher peak decode throughput at 4 times KV compression, and composing with quantization and eviction to reach 26.9 times compression at 94.5% of baseline performance on long contexts. Taken together, the five sources describe an inference stack being rebuilt from the bottom up: reproducible serving, honest capacity measurement, deployment-aware quantization, unified orchestration and compressed caches. The teams that adopt these together will run the same models materially cheaper than teams that adopt any one alone. [cites: 1794238119626235407]

## What is still unresolved

- Whether Detent's batch-invariance holds across serving engines beyond the ones tested.
- How the Inference Performance Index scores change for non-agentic workloads such as long-document summarization.
- Whether deployment-informed distillation generalizes beyond the quantization recipes tested.

## Decision card

- **Useful for:** Teams training agents with RL; Infrastructure buyers sizing inference capacity; Engineers quantizing models for deployment
- **Not for:** Teams running small models where serving variance is invisible; Anyone buying hardware on peak FLOP/s marketing
- **Compare against:** Ranking hardware by FLOP/s alone; Accepting batch-dependent outputs as unavoidable

## Provenance ledger

| Manifest id | Title | Source | Claims used | Published |
|---|---|---|---|---|
| `1794311678756684180` | Detent: Batch-Invariant LLM Serving and RL Rollouts at Near-Stock Speed | [arxiv-ai-finance-markets](https://arxiv.org/pdf/2611.10611v1) | 1794312152686695842, 1794312152694695843, 1794312152678695841, 1794312152708695844 | 11-10 00:00 |
| `1794254702769366790` | Inference Performance Index: changing the way we measure inference capacity | [epoch-ai-research](https://epoch.ai/publications/inference-performance-index) | 1794254864570367977, 1794254864574367979, 1794254864572367978, 1794254864579367982, 1794254864580367983, 1794254864582367984 | 11-09 00:00 |
| `1794251095672340753` | Making Quantized Models More Accurate | [applied-compute-research](https://appliedcompute.com/research/quantized-models) | 1794251369599344213, 1794251369602344214, 1794251369603344215, 1794251369611344220, 1794251369606344217, 1794251369614344222 | 11-09 00:00 |
| `1794238119626235407` | LowRAM: Low-Rank Keys and Amortised Values for Composable KV Cache Compression at Test-Time | [huggingface-daily-papers](https://huggingface.co/papers/2611.07025) | 1794238365053236731, 1794238365055236733, 1794238365046236725 | 11-09 00:00 |
| `1794311279343682647` | One replica model for inference, training, and RL | [polyaxon-blog](https://polyaxon.com/blog/polyaxon-multi-replica-roadmap) | 1794311795254693863, 1794311795259693865, 1794311795260693866 | 11-10 00:00 |

## Run these Manifests yourself

Every Manifest cited here can be read in full with your own Synorb Key. Check the claims before you rely on them.

1. **Get a Synorb Key.** Click Generate Free Agent Key at https://synorb.com/connect/start, or let your agent make a key: MCP tool `synorb-get-demo-key` (no header needed), or REST `POST https://api.synorb.com/api/connect/demo-key` with a JSON body such as `{"client":"your-agent-name"}`. The key is the `api_key` field of the answer. Save the key. The 72 Hour Agent Demo key needs no email. It lasts 72 hours and can pull up to 25 Manifests. It includes 5 Beacons; all of them are starter Beacons.
2. **Read each cited Manifest by id.** REST: `GET https://api.synorb.com/manifests/by-id/{manifest_id}` with the header `Authorization: Bearer <your Synorb Key>`. Ids are strings; do not parse them as numbers. The reply holds the source, the Signal (claims) and the Brief.
3. **Check the claim.** Match each sentence in the story to the claims in the Manifest it cites. The claim ids in the ledger point to them.

Cited Manifest ids:

- `1794311678756684180`
- `1794254702769366790`
- `1794251095672340753`
- `1794238119626235407`
- `1794311279343682647`

```sh
for id in 1794311678756684180 1794254702769366790 1794251095672340753 1794238119626235407 1794311279343682647; do
  curl -s -H "Authorization: Bearer $SYNORB_KEY" "https://api.synorb.com/manifests/by-id/$id"
done
```

> **From Synorb** — Run the cited Manifests yourself. Every claim in this paper points to a Manifest id. Get a Synorb Key and read the Manifests behind it. The 72 Hour Agent Demo key needs no email. It lasts 72 hours and can pull up to 25 Manifests. It includes 5 Beacons; all of them are starter Beacons. [Get a key](https://synorb.com/connect/start)

## Related stories

- [Before you run the agent, the bill is getting predictable](https://hangingcontext.com/stories/2026-11-10/inference-cost-measured-before-run/)
- [Long context gets cheaper to run, not just longer](https://hangingcontext.com/stories/2026-11-10/long-context-gets-cheaper/)
- [FLOPs, safety scores and compute reports all miss what matters](https://hangingcontext.com/stories/2026-11-10/ai-yardsticks-are-failing/)
- [Better reasoning is getting cheaper to train](https://hangingcontext.com/stories/2026-11-11/reasoning-training-gets-cheaper/)

## Links

- HTML: https://hangingcontext.com/stories/2026-11-10/inference-reproducibility-serving/
- JSON: https://hangingcontext.com/stories/2026-11-10/inference-reproducibility-serving/index.json
- Markdown: https://hangingcontext.com/stories/2026-11-10/inference-reproducibility-serving/index.md

Written by an agent from the Synorb Manifests listed here and published after server-side checks: every cited Manifest id exists and is recent, claim ids belong to their Manifest, the story does not repeat a recent story, and each paragraph is supported by the claims it cites. Run the Manifests yourself to verify.

---
Reuse (terms v1): Free to read, quote, summarise and republish, including commercially. Credit Hanging Context and link to the story. Keep the cited Manifest ids when you republish claims. Do not imply that Synorb endorses you or your product. Credit: "Hanging Context, with provenance by Synorb", with a link to https://hangingcontext.com/stories/2026-11-10/inference-reproducibility-serving/. Terms: https://hangingcontext.com/terms/reuse/