Throughput is the new model spec, and the gap between hosts is now fortyfold

The same hardware, a different result

BLACKBOX AI reports that its Kimi K3.1 Pro model achieves significantly higher throughput and lower latency than Fireworks AI on identical eight-GPU NVIDIA B300 hardware, and the numbers are stark enough to change how teams should shop for inference. On a single node with eight NVIDIA B300 GPUs, BLACKBOX AI served Kimi K3.1 Pro at 20.23 million total tokens per minute, while the Fireworks AI dedicated deployment reached a peak of 5.66 million tokens per minute on the same GPU count and type. BLACKBOX AI says it achieved 3.6 times the peak throughput of Fireworks AI for the same coding workload. The hardware is the same. The model is the same class of workload. The difference is the serving stack. m:…0784

The gap widens as concurrency increases, which is the part that matters for real deployments. BLACKBOX AI says its throughput rose from 5.14 million tokens per minute at 8 conversations to 20.23 million at 48 conversations, while Fireworks AI throughput peaked at 16 concurrent conversations. The throughput gap reached 42.1 times at 48 concurrent conversations, and at 32 concurrent conversations Fireworks AI throughput fell 77 percent to 1.33 million tokens per minute. At 64 concurrent conversations, BLACKBOX AI served 19.19 million tokens per minute, approximately 5 percent below its peak. One stack scales with load; the other falls over. m:…0784

Latency tells the same story from the user's side. At 32 concurrent coding conversations, the median wait for a first token was 0.52 seconds on BLACKBOX AI and 51.50 seconds on Fireworks AI. Once generation began, median output speed was 41 tokens per second per request for BLACKBOX AI versus 4 tokens per second for Fireworks AI. A coding agent that waits 51.50 seconds for its first token is not a slow tool; it is a stalled one, and the developer waiting on it is paying for the stall in lost flow as well as tokens. m:…0784

A new flagship at a commodity price

DeepSeek V4.1 Flash is now available on Verda Inference, priced at 0.15 dollars per million input tokens and 0.60 dollars per million output tokens, with a 1M-token context window. Cached input tokens are priced at 0.006 dollars per million. Those are the numbers that will decide whether the model gets used: a million-token context window at a price where stuffing the whole codebase into the prompt stops being a luxury. m:…7528

The architecture explains the price. DeepSeek V4.1 Flash is a 552B-parameter Mixture-of-Experts model that activates 8B parameters while reading input and 16B parameters while writing output, and maintains a KV cache of 890 bytes per token. Verda Inference uses the SGLang engine to serve the model, and its deployment achieves output speeds peaking at approximately 29,300 tokens per second per GPU. Verda provides an OpenAI-compatible API, so switching costs are low. The model is big on paper and small in practice, which is the whole point of the mixture-of-experts design. m:…7528

The biggest models no longer fit in one box

Nvidia's developer blog addresses the other end of the same problem: what to do when the model does not fit at all. Frontier-scale models require keeping all weights in GPU memory, and the largest models no longer fit on a single machine. That is a sentence that would have sounded absurd a few years ago and is now a routine engineering constraint. m:…1810

The answer Nvidia documents is to serve GLM-5.3 across two Nvidia DGX Stations connected over ConnectX-8 RDMA, using vLLM tensor parallelism to split model weights and computation across GPUs. GLM-5.3 NVFP4 is a quantized version of Z.ai's mixture-of-experts model. The implication for buyers is straightforward: the unit of inference is no longer a GPU or even a server, but a connected cluster, and the vendors who can make the cluster behave like one machine will set the price of intelligence. m:…1810

Put the three stories together and the pattern is clear. The throughput gap between hosts reached 42.1 times at 48 concurrent conversations, and Verda reports output speeds peaking at approximately 29,300 tokens per second per GPU. Model quality is converging; serving quality is diverging. The teams that benchmark the serving stack at their own concurrency, rather than trusting a headline peak, will pay less and wait less. The ones that do not will discover the gap in production, one stalled agent at a time. m:…0784 m:…7528