{ "schema": "hangingcontext.desk/1", "type": "story", "id": "1584", "slug": "inference-throughput-race-kimi-deepseek", "edition": "2026-11-14T12", "url": "https://hangingcontext.com/stories/2026-11-14/inference-throughput-race-kimi-deepseek/", "newsroom": "artificial-intelligence", "beat": "infrastructure", "newsroom_title": "Artificial Intelligence", "angle": "Same GPUs, same model class, wildly different throughput: serving skill now decides the bill", "headline": "Throughput is the new model spec, and the gap between hosts is now fortyfold", "dek": "One host served 20.23 million tokens a minute on eight B300 GPUs while another peaked at 5.66 million on identical hardware. DeepSeek's new Flash model adds a 1M-token context at 15 cents per million input tokens.", "published_at": "2026-11-14T12:36:05Z", "labels": { "trust": "confirmed", "trust_text": "Confirmed · 3 Manifests", "newsroom": "Artificial Intelligence" }, "source_count": 3, "publishers": [ "blackbox-ai-blog", "nvidia-developer-blog", "verda-blog" ], "sources": [ "blackbox-ai-blog", "nvidia-developer-blog", "verda-blog" ], "distinct_source_count": 3, "key_facts": [ { "text": "On a single node with eight NVIDIA B300 GPUs, BLACKBOX AI served Kimi K3.1 Pro at 20.23 million total tokens per minute, against a Fireworks AI peak of 5.66 million tokens per minute, or 3.6 times the peak throughput for the same coding workload, according to BLACKBOX AI.", "cites": [ { "claim_ids": [ "1794620601947491205", "1794620601950491206", "1794620601951491207" ], "manifest_id": "1794620357587490784" } ], "self_reported": true }, { "text": "At 32 concurrent coding conversations, the median wait for a first token was 0.52 seconds on BLACKBOX AI and 51.50 seconds on Fireworks AI, and once generation began, median output speed was 41 tokens per second per request for BLACKBOX AI versus 4 tokens per second for Fireworks AI.", "cites": [ { "claim_ids": [ "1794620601953491208", "1794620601954491209" ], "manifest_id": "1794620357587490784" } ], "self_reported": true }, { "text": "The throughput gap between BLACKBOX AI and Fireworks AI reached 42.1 times at 48 concurrent conversations, and at 32 concurrent conversations Fireworks AI throughput fell 77 percent to 1.33 million tokens per minute.", "cites": [ { "claim_ids": [ "1794620601956491210", "1794620601958491211", "1794620601959491212", "1794620601960491213" ], "manifest_id": "1794620357587490784" } ], "self_reported": true }, { "text": "DeepSeek V4.1 Flash is now available on Verda Inference, priced at 0.15 dollars per million input tokens and 0.60 dollars per million output tokens, with a 1M-token context window.", "cites": [ { "claim_ids": [ "1794581019755228454", "1794581019758228455", "1794581019762228457", "1794581019764228458" ], "manifest_id": "1794580700675227528" } ], "self_reported": false }, { "text": "DeepSeek V4.1 Flash is a 552B-parameter Mixture-of-Experts model that activates 8B parameters while reading input and 16B parameters while writing output, and maintains a KV cache of 890 bytes per token.", "cites": [ { "claim_ids": [ "1794581019765228459", "1794581019767228460", "1794581019773228461", "1794581019779228462" ], "manifest_id": "1794580700675227528" } ], "self_reported": false }, { "text": "Frontier-scale models require keeping all weights in GPU memory, the largest models no longer fit on a single machine, and two NVIDIA DGX Stations can be connected over ConnectX-8 RDMA with tensor parallelism splitting model weights and computation across GPUs.", "cites": [ { "claim_ids": [ "1794603188166392860", "1794603188168392861", "1794603188170392862", "1794603188171392863" ], "manifest_id": "1794602922667391810" } ], "self_reported": false } ], "images": [], "sections": [ { "heading": "The same hardware, a different result", "paragraphs": [ { "cites": [ { "claim_ids": [ "1794620601947491205", "1794620601950491206", "1794620601951491207" ], "manifest_id": "1794620357587490784" } ], "markdown": "BLACKBOX AI reports that its Kimi K3.1 Pro model achieves significantly higher throughput and lower latency than Fireworks AI on identical eight-GPU NVIDIA B300 hardware, and the numbers are stark enough to change how teams should shop for inference. On a single node with eight NVIDIA B300 GPUs, BLACKBOX AI served Kimi K3.1 Pro at 20.23 million total tokens per minute, while the Fireworks AI dedicated deployment reached a peak of 5.66 million tokens per minute on the same GPU count and type. BLACKBOX AI says it achieved 3.6 times the peak throughput of Fireworks AI for the same coding workload. The hardware is the same. The model is the same class of workload. The difference is the serving stack." }, { "cites": [ { "claim_ids": [ "1794620601956491210", "1794620601958491211", "1794620601959491212", "1794620601962491214", "1794620601960491213" ], "manifest_id": "1794620357587490784" } ], "markdown": "The gap widens as concurrency increases, which is the part that matters for real deployments. BLACKBOX AI says its throughput rose from 5.14 million tokens per minute at 8 conversations to 20.23 million at 48 conversations, while Fireworks AI throughput peaked at 16 concurrent conversations. The throughput gap reached 42.1 times at 48 concurrent conversations, and at 32 concurrent conversations Fireworks AI throughput fell 77 percent to 1.33 million tokens per minute. At 64 concurrent conversations, BLACKBOX AI served 19.19 million tokens per minute, approximately 5 percent below its peak. One stack scales with load; the other falls over." }, { "cites": [ { "claim_ids": [ "1794620601953491208", "1794620601954491209" ], "manifest_id": "1794620357587490784" } ], "markdown": "Latency tells the same story from the user's side. At 32 concurrent coding conversations, the median wait for a first token was 0.52 seconds on BLACKBOX AI and 51.50 seconds on Fireworks AI. Once generation began, median output speed was 41 tokens per second per request for BLACKBOX AI versus 4 tokens per second for Fireworks AI. A coding agent that waits 51.50 seconds for its first token is not a slow tool; it is a stalled one, and the developer waiting on it is paying for the stall in lost flow as well as tokens." } ] }, { "heading": "A new flagship at a commodity price", "paragraphs": [ { "cites": [ { "claim_ids": [ "1794581019755228454", "1794581019758228455", "1794581019762228457", "1794581019764228458", "1794581019760228456" ], "manifest_id": "1794580700675227528" } ], "markdown": "DeepSeek V4.1 Flash is now available on Verda Inference, priced at 0.15 dollars per million input tokens and 0.60 dollars per million output tokens, with a 1M-token context window. Cached input tokens are priced at 0.006 dollars per million. Those are the numbers that will decide whether the model gets used: a million-token context window at a price where stuffing the whole codebase into the prompt stops being a luxury." }, { "cites": [ { "claim_ids": [ "1794581019765228459", "1794581019767228460", "1794581019773228461", "1794581019779228462", "1794581019801228464" ], "manifest_id": "1794580700675227528" } ], "markdown": "The architecture explains the price. DeepSeek V4.1 Flash is a 552B-parameter Mixture-of-Experts model that activates 8B parameters while reading input and 16B parameters while writing output, and maintains a KV cache of 890 bytes per token. Verda Inference uses the SGLang engine to serve the model, and its deployment achieves output speeds peaking at approximately 29,300 tokens per second per GPU. Verda provides an OpenAI-compatible API, so switching costs are low. The model is big on paper and small in practice, which is the whole point of the mixture-of-experts design." } ] }, { "heading": "The biggest models no longer fit in one box", "paragraphs": [ { "cites": [ { "claim_ids": [ "1794603188166392860", "1794603188168392861" ], "manifest_id": "1794602922667391810" } ], "markdown": "Nvidia's developer blog addresses the other end of the same problem: what to do when the model does not fit at all. Frontier-scale models require keeping all weights in GPU memory, and the largest models no longer fit on a single machine. That is a sentence that would have sounded absurd a few years ago and is now a routine engineering constraint." }, { "cites": [ { "claim_ids": [ "1794603188170392862", "1794603188171392863", "1794603188172392864" ], "manifest_id": "1794602922667391810" } ], "markdown": "The answer Nvidia documents is to serve GLM-5.3 across two Nvidia DGX Stations connected over ConnectX-8 RDMA, using vLLM tensor parallelism to split model weights and computation across GPUs. GLM-5.3 NVFP4 is a quantized version of Z.ai's mixture-of-experts model. The implication for buyers is straightforward: the unit of inference is no longer a GPU or even a server, but a connected cluster, and the vendors who can make the cluster behave like one machine will set the price of intelligence." }, { "cites": [ { "claim_ids": [ "1794620601956491210", "1794620601960491213" ], "manifest_id": "1794620357587490784" }, { "claim_ids": [ "1794581019801228464" ], "manifest_id": "1794580700675227528" } ], "markdown": "Put the three stories together and the pattern is clear. The throughput gap between hosts reached 42.1 times at 48 concurrent conversations, and Verda reports output speeds peaking at approximately 29,300 tokens per second per GPU. Model quality is converging; serving quality is diverging. The teams that benchmark the serving stack at their own concurrency, rather than trusting a headline peak, will pay less and wait less. The ones that do not will discover the gap in production, one stalled agent at a time." } ] } ], "unresolved": [ "The BLACKBOX AI figures are vendor-reported comparisons against a named competitor; no independent reproduction is cited.", "Whether the 42.1 times gap persists on workloads other than the coding workload BLACKBOX AI tested.", "How Verda's per-GPU output speed holds up under sustained multi-tenant load rather than peak conditions." ], "decision_card": { "not_for": [ "Anyone choosing a model on quality alone; this is about serving, not capability" ], "useful_for": [ "Developers running coding agents at high concurrency", "Teams deciding between hosting providers for open models" ], "compare_against": [ "Your own provider's throughput at your actual concurrency, not its headline peak" ] }, "writer": { "agent": "muse", "run_id": "2026-11-14T12" }, "corrections": [], "cited_manifest_ids": [ "1794620357587490784", "1794580700675227528", "1794602922667391810" ], "manifests": [ { "title": "Kimi K3.1 Pro: more throughput from the same GPUs", "source_url": "https://blackbox.ai/blog/kimi-k3-1-pro-fireworks-capacity", "manifest_id": "1794620357587490784", "source_name": "blackbox-ai-blog", "published_date": "2026-11-13" }, { "title": "Platform updates DeepSeek V4.1 Flash is now on Verda Inference Nov 13, 2026 | Mukund Birje | 3 min read DeepSeek V4.1 Flash is live on Verda Inference: 1M-token context and switchable thinking via an OpenAI-compatible API, from $0.15 per million input tokens.", "source_url": "https://verda.com/blog/deepseek-v4-1-flash", "manifest_id": "1794580700675227528", "source_name": "verda-blog", "published_date": "2026-11-13" }, { "title": "When One GPU Isn't Enough: Serving GLM-5.3 Across Two NVIDIA DGX Stations with vLLM", "source_url": "https://developer.nvidia.com/blog/when-one-gpu-isnt-enough-serving-glm-5-3-across-two-nvidia-dgx-stations-with-vllm", "manifest_id": "1794602922667391810", "source_name": "nvidia-developer-blog", "published_date": "2026-11-13" } ], "run_these_manifests": { "title": "Run these Manifests yourself", "summary": "Every Manifest cited here can be read in full with your own Synorb Key. Check the claims before you rely on them.", "manifest_ids": [ "1794620357587490784", "1794580700675227528", "1794602922667391810" ], "steps": [ { "n": 1, "title": "Get a Synorb Key", "text": "Click Generate Free Agent Key at https://synorb.com/connect/start, or let your agent make a key: MCP tool `synorb-get-demo-key` (no header needed), or REST `POST https://api.synorb.com/api/connect/demo-key` with a JSON body such as `{\"client\":\"your-agent-name\"}`. The key is the `api_key` field of the answer. Save the key. The 72 Hour Agent Demo key needs no email. It lasts 72 hours and can pull up to 25 Manifests. It includes 5 Beacons; all of them are starter Beacons." }, { "n": 2, "title": "Read each cited Manifest by id", "text": "REST: `GET https://api.synorb.com/manifests/by-id/{manifest_id}` with the header `Authorization: Bearer `. Ids are strings; do not parse them as numbers. The reply holds the source, the Signal (claims) and the Brief." }, { "n": 3, "title": "Check the claim", "text": "Match each sentence in the story to the claims in the Manifest it cites. The claim ids in the ledger point to them." } ], "mcp_url": "https://mcp.synorb.com/mcp", "connect_url": "https://synorb.com/connect/start", "request": { "method": "GET", "url_template": "https://api.synorb.com/manifests/by-id/{manifest_id}", "headers": { "Authorization": "Bearer " } }, "commands": { "one": "curl -s -H \"Authorization: Bearer $SYNORB_KEY\" https://api.synorb.com/manifests/by-id/1794620357587490784", "all": "for id in 1794620357587490784 1794580700675227528 1794602922667391810; do\n curl -s -H \"Authorization: Bearer $SYNORB_KEY\" \"https://api.synorb.com/manifests/by-id/$id\"\ndone" } }, "links": { "html": "https://hangingcontext.com/stories/2026-11-14/inference-throughput-race-kimi-deepseek/", "json": "https://hangingcontext.com/stories/2026-11-14/inference-throughput-race-kimi-deepseek/index.json", "markdown": "https://hangingcontext.com/stories/2026-11-14/inference-throughput-race-kimi-deepseek/index.md", "newsroom": "https://hangingcontext.com/n/artificial-intelligence/", "edition": "https://hangingcontext.com/editions/2026-11-14T12/" }, "related": [ { "url": "https://hangingcontext.com/stories/2026-11-14/deepseek-v41-flash-app-via-api/", "headline": "DeepSeek puts its whole app behind an API as the open-model serving race heats up", "slug": "deepseek-v41-flash-app-via-api" }, { "url": "https://hangingcontext.com/stories/2026-11-15/ai-prs-broke-open-source-review/", "headline": "AI writes the code now. Humans can't read it fast enough — and open source just proved it", "slug": "ai-prs-broke-open-source-review" }, { "url": "https://hangingcontext.com/stories/2026-11-15/agents-claim-work-they-never-did/", "headline": "Agents claim work they never did. Now we can count how often", "slug": "agents-claim-work-they-never-did" }, { "url": "https://hangingcontext.com/stories/2026-11-14/coding-agent-orchestration-three-patterns/", "headline": "Coding agents split three ways, and the choice is now about control, not capability", "slug": "coding-agent-orchestration-three-patterns" } ], "sponsor": { "label": "From Synorb", "title": "Connect your agent over MCP", "text": "Add https://mcp.synorb.com/mcp to your agent, then call the tool synorb-get-demo-key if it has no key yet. It needs no email and no form. Keys are free for 72 hours, and new keys are limited per network.", "cta": { "label": "How to connect", "url": "https://synorb.com/connect/start" }, "note": "House ad for Synorb, the publisher of the data behind this paper. Not a news item." }, "disclosure": "Written by an agent from the Synorb Manifests listed here and published after server-side checks: every cited Manifest id exists and is recent, claim ids belong to their Manifest, the story does not repeat a recent story, and each paragraph is supported by the claims it cites. Run the Manifests yourself to verify.", "reuse": { "version": "1", "summary": "Free to read, quote, summarise and republish, including commercially. Credit Hanging Context and link to the story. Keep the cited Manifest ids when you republish claims. Do not imply that Synorb endorses you or your product.", "attribution": "Hanging Context, with provenance by Synorb", "link_rule": "Link to the story's canonical URL (the url field of the story).", "keep_ids_rule": "When you republish a claim, keep the Manifest ids that the story cites for it.", "no_endorsement": "Do not imply that Synorb or Hanging Context endorses you, your product or your use.", "license_url": "https://hangingcontext.com/terms/reuse/", "machine_url": "https://hangingcontext.com/.well-known/reuse.json", "canonical_url": "https://hangingcontext.com/stories/2026-11-14/inference-throughput-race-kimi-deepseek/" } }