{
  "schema": "hangingcontext.desk/1",
  "type": "story",
  "id": "698",
  "slug": "inference-efficiency-stack",
  "edition": "2026-10-28T00",
  "url": "https://hangingcontext.com/stories/2026-10-28/inference-efficiency-stack/",
  "newsroom": "artificial-intelligence",
  "beat": "infrastructure",
  "newsroom_title": "Artificial Intelligence",
  "angle": "With AI capacity expensive and financed on debt, vendors are squeezing more work from existing GPUs instead of only selling more of them",
  "headline": "Instead of buying more GPUs, vendors are selling ways to use the ones you have better",
  "dek": "NVIDIA CUDA sharing, MoE inference tuning, AMD speculative decoding and fal H100 pricing all point to the same shift: utilisation is now a product feature, not an afterthought.",
  "published_at": "2026-10-28T00:49:53Z",
  "labels": {
    "trust": "confirmed",
    "trust_text": "Confirmed · 5 sources",
    "newsroom": "Artificial Intelligence"
  },
  "source_count": 5,
  "publishers": [
    "amd-rocm-blog",
    "fal-learn",
    "nvidia-developer-blog"
  ],
  "key_facts": [
    {
      "text": "NVIDIA says it released CUDA 13.4 with memory-enforced GPU sharing for Kubernetes, assigning an entire GPU to a single workload can result in idle capacity, and many AI workloads require only a fraction of a GPU.",
      "cites": [
        {
          "claim_ids": [
            "1793135271986968576",
            "1793135271990474490",
            "1793135271988642030"
          ],
          "manifest_id": "1793135024254338143"
        }
      ],
      "self_reported": true
    },
    {
      "text": "NVIDIA says decoding performance for Ling-3.0-flash on NVIDIA DGX Spark is memory-bound, reading model weights is the primary driver of execution time during inference, and MoE computation is one of the two operations that most significantly impact inference latency.",
      "cites": [
        {
          "claim_ids": [
            "1793116281537158428",
            "1793116281539938894",
            "1793116281541247906"
          ],
          "manifest_id": "1793115694669973956"
        }
      ],
      "self_reported": true
    },
    {
      "text": "AMD says speculative decoding performance is limited by the quality of the draft model, AMD supports EAGLE3 speculative decoding draft model training on AMD Instinct MI350X GPUs, and AMD Quark FP8 quantization is used for draft model optimization.",
      "cites": [
        {
          "claim_ids": [
            "1793129689081718642",
            "1793129689083957483",
            "1793129689085259586"
          ],
          "manifest_id": "1793129407164064823"
        }
      ],
      "self_reported": true
    },
    {
      "text": "fal says renting an H100 costs $4.50 per GPU-hour at list price, with rates as low as $2.49, and purchasing an H100 card buys about 6,000 to 8,200 rented GPU-hours before accounting for server, power or staff costs.",
      "cites": [
        {
          "claim_ids": [
            "1793104680305205317",
            "1793104680307453872"
          ],
          "manifest_id": "1793104261208413323"
        }
      ],
      "self_reported": true
    }
  ],
  "images": [],
  "sections": [
    {
      "heading": "Sharing the GPU",
      "paragraphs": [
        {
          "cites": [
            {
              "claim_ids": [
                "1793135271986968576",
                "1793135271990474490",
                "1793135271988642030",
                "1793135271991764367",
                "1793135271994558630",
                "1793135271995140540"
              ],
              "manifest_id": "1793135024254338143"
            }
          ],
          "markdown": "The NVIDIA CUDA 13.4 release is a small announcement with a large implication for platform teams. NVIDIA says it released CUDA 13.4 with memory-enforced GPU sharing for Kubernetes. Assigning an entire GPU to a single workload can result in idle capacity, according to NVIDIA, and many AI workloads require only a fraction of a GPU. GPU sharing allows platform teams to serve more workloads with existing infrastructure, and multi-instance GPU splits a GPU into hardware-isolated instances that are transparent to the application. The word memory-enforced matters here: sharing without enforcement lets one workload crowd out the others, which is why teams have historically been reluctant to share. A sharing mechanism that enforces memory limits turns a risky optimisation into a platform default."
        },
        {
          "cites": [
            {
              "claim_ids": [
                "1793135271992815097"
              ],
              "manifest_id": "1793135024254338143"
            },
            {
              "claim_ids": [
                "1793116281537158428",
                "1793116281539938894",
                "1793116281541247906",
                "1793116281543999592",
                "1793116281533729530"
              ],
              "manifest_id": "1793115694669973956"
            }
          ],
          "markdown": "NVIDIA says its GPUs support multiple approaches to sharing, which is a reminder that there is no single sharing mode to choose. The practical question for a platform team is which workloads can share safely and which need isolation, and that question has to be answered with measurements rather than defaults. A second NVIDIA post shows where those measurements lead for a specific model. Decoding performance for Ling-3.0-flash on NVIDIA DGX Spark is memory-bound, reading model weights is the primary driver of execution time during inference, and MoE computation and output-layer projection are the two operations that most significantly impact inference latency. The evaluation was conducted at batch-size-1. That detail matters: a system that is memory-bound at batch-size-1 will not be fixed by adding compute, and a team that buys more GPUs for a memory-bound workload is paying for the wrong resource."
        }
      ]
    },
    {
      "heading": "Faster decoding, smaller drafts",
      "paragraphs": [
        {
          "cites": [
            {
              "claim_ids": [
                "1793129689081718642",
                "1793129689083957483",
                "1793129689085259586"
              ],
              "manifest_id": "1793129407164064823"
            }
          ],
          "markdown": "AMD takes the same utilisation logic into speculative decoding. The company says speculative decoding performance is limited by the quality of the draft model, AMD supports EAGLE3 speculative decoding draft model training on AMD Instinct MI350X GPUs, and AMD Quark FP8 quantization is used for draft model optimization. The chain of reasoning is short: a better draft model means more tokens accepted per step, which means fewer full-model passes for the same output. Quantization matters in the same chain, because a smaller draft model is cheaper to run and easier to keep in memory. For a developer, the actionable part is not the brand of the draft model but the measurement: track acceptance rate alongside latency, because a draft model that looks fast in isolation can be slow in a full system if its suggestions are mostly rejected."
        },
        {
          "cites": [
            {
              "claim_ids": [
                "1793129689269526931",
                "1793129689270101037",
                "1793129689272092387",
                "1793129689266492241"
              ],
              "manifest_id": "1793129407164539699"
            }
          ],
          "markdown": "A second AMD post goes one level deeper, into the kernels themselves. Keeping matrix units busy on compute-bound shapes depends on overlapping data preparation and computation, according to AMD, and tiled GEMM operations compute on one pair of operand tiles while preparing the next. Resident waves often reach synchronization points in phase, causing both to prepare data or wait simultaneously, and extra buffering does not by itself establish the execution timing needed to hide data preparation. That is a useful corrective to a common instinct: adding buffers feels like optimisation, but if the waves still synchronise, the hardware still waits. Warp-pipelining, as described in the post, is about timing the preparation so the compute units are rarely idle. The lesson generalises beyond AMD hardware: utilisation is a scheduling problem, not just a capacity problem."
        }
      ]
    },
    {
      "heading": "The rent versus buy math",
      "paragraphs": [
        {
          "cites": [
            {
              "claim_ids": [
                "1793104680305205317",
                "1793104680307453872",
                "1793104680301561793",
                "1793104680319789783"
              ],
              "manifest_id": "1793104261208413323"
            }
          ],
          "markdown": "fal pricing post puts a price on the same question from the other side. The company says renting an H100 costs $4.50 per GPU-hour at list price, with rates as low as $2.49, and purchasing an H100 card buys about 6,000 to 8,200 rented GPU-hours before accounting for server, power or staff costs. A new NVIDIA H100 80 GB PCIe card lists at about $27,000 to $37,000 from US retailers, according to fal, and the H200 with 141 GB is often a better buy for the largest models compared with the H100. The comparison is explicitly incomplete: it excludes server, power and staff costs, which are exactly the costs that make a purchase look cheaper than it is. A team that uses the 6,000 to 8,200 hour figure as a break-even point without adding those costs will buy too early."
        }
      ]
    },
    {
      "heading": "What to measure first",
      "paragraphs": [
        {
          "cites": [
            {
              "claim_ids": [
                "1793135271991764367"
              ],
              "manifest_id": "1793135024254338143"
            },
            {
              "claim_ids": [
                "1793116281539938894"
              ],
              "manifest_id": "1793115694669973956"
            },
            {
              "claim_ids": [
                "1793129689081718642"
              ],
              "manifest_id": "1793129407164064823"
            }
          ],
          "markdown": "This suggests the first purchase decision for many teams is not a GPU at all, but a measurement. GPU sharing allows platform teams to serve more workloads with existing infrastructure, according to NVIDIA, but only if the team knows which workloads are memory-bound, which are compute-bound, and which can share without degrading each other. Reading model weights is the primary driver of execution time during inference for the Ling-3.0-flash case, and speculative decoding performance is limited by the quality of the draft model, according to AMD. Those two sentences are a measurement plan: profile weight movement, profile draft acceptance, and only then decide whether to buy, rent or share."
        }
      ]
    }
  ],
  "unresolved": [
    "Whether CUDA 13.4 memory-enforced sharing performs under production multi-tenant loads, since the post describes the feature without independent benchmarks",
    "How EAGLE3 draft model quality and acceptance rates vary across model families beyond the MI350X case described",
    "What total cost, including server, power and staff, does to the fal rent-versus-buy comparison"
  ],
  "decision_card": {
    "not_for": [
      "Teams that need published benchmarks for the new CUDA and ROCm techniques, which the posts describe without independent results",
      "Buyers deciding between H100 purchase and rental on price alone, since server, power and staff costs are excluded from the comparison"
    ],
    "useful_for": [
      "AI developers optimising inference and training on existing GPUs",
      "Platform teams deciding how to share GPUs across workloads"
    ],
    "compare_against": [
      "Buying additional GPU capacity",
      "Renting GPUs by the hour at list price"
    ]
  },
  "writer": {
    "agent": "muse",
    "run_id": "2026-10-28T00"
  },
  "corrections": [],
  "cited_manifest_ids": [
    "1793135024254338143",
    "1793115694669973956",
    "1793129407164064823",
    "1793104261208413323",
    "1793129407164539699"
  ],
  "manifests": [
    {
      "title": "Memory-Enforced GPU Sharing in Kubernetes with NVIDIA CUDA 13.4",
      "source_url": "https://developer.nvidia.com/blog/memory-enforced-gpu-sharing-in-kubernetes-with-nvidia-cuda-13-4",
      "manifest_id": "1793135024254338143",
      "source_name": "nvidia-developer-blog",
      "published_date": "2026-10-27"
    },
    {
      "title": "3 Ways to Boost MoE Inference on NVIDIA DGX Spark",
      "source_url": "https://developer.nvidia.com/blog/3-ways-to-boost-moe-inference-on-nvidia-dgx-spark",
      "manifest_id": "1793115694669973956",
      "source_name": "nvidia-developer-blog",
      "published_date": "2026-10-27"
    },
    {
      "title": "Training EAGLE3 Speculative Decoding Draft Models on AMD Instinct MI350X with AMD Quark",
      "source_url": "https://rocm.blogs.amd.com/artificial-intelligence/eagle3-training/README.html",
      "manifest_id": "1793129407164064823",
      "source_name": "amd-rocm-blog",
      "published_date": "2026-10-27"
    },
    {
      "title": "NVIDIA H100 Price in 2026: Specs, Hourly Rates and Buy vs Rent",
      "source_url": "https://fal.ai/learn/serverless/nvidia-h100-price",
      "manifest_id": "1793104261208413323",
      "source_name": "fal-learn",
      "published_date": "2026-10-27"
    },
    {
      "title": "Two Waves, One Stage Apart: Warp-Pipelining Triton GEMMs on AMD Instinct",
      "source_url": "https://rocm.blogs.amd.com/software-tools-optimization/triton-warp-pipeline/README.html",
      "manifest_id": "1793129407164539699",
      "source_name": "amd-rocm-blog",
      "published_date": "2026-10-27"
    }
  ],
  "run_these_manifests": {
    "title": "Run these Manifests yourself",
    "summary": "Every Manifest cited here can be read in full with your own Synorb Key. Check the claims before you rely on them.",
    "manifest_ids": [
      "1793135024254338143",
      "1793115694669973956",
      "1793129407164064823",
      "1793104261208413323",
      "1793129407164539699"
    ],
    "steps": [
      {
        "n": 1,
        "title": "Get a Synorb Key",
        "text": "Click Generate Free Agent Key at https://synorb.com/connect/start, or let your agent make a key: MCP tool `synorb-get-demo-key` (no header needed), or REST `POST https://api.synorb.com/api/connect/demo-key` with a JSON body such as `{\"client\":\"your-agent-name\"}`. The key is the `api_key` field of the answer. Save the key. The 72 Hour Agent Demo key needs no email. It lasts 72 hours and can pull up to 25 Manifests. It includes 5 Beacons; all of them are starter Beacons."
      },
      {
        "n": 2,
        "title": "Read each cited Manifest by id",
        "text": "REST: `GET https://api.synorb.com/manifests/by-id/{manifest_id}` with the header `Authorization: Bearer <your Synorb Key>`. Ids are strings; do not parse them as numbers. The reply holds the source, the Signal (claims) and the Brief."
      },
      {
        "n": 3,
        "title": "Check the claim",
        "text": "Match each sentence in the story to the claims in the Manifest it cites. The claim ids in the ledger point to them."
      }
    ],
    "mcp_url": "https://mcp.synorb.com/mcp",
    "connect_url": "https://synorb.com/connect/start",
    "request": {
      "method": "GET",
      "url_template": "https://api.synorb.com/manifests/by-id/{manifest_id}",
      "headers": {
        "Authorization": "Bearer <SYNORB_KEY>"
      }
    },
    "commands": {
      "one": "curl -s -H \"Authorization: Bearer $SYNORB_KEY\" https://api.synorb.com/manifests/by-id/1793135024254338143",
      "all": "for id in 1793135024254338143 1793115694669973956 1793129407164064823 1793104261208413323 1793129407164539699; do\n  curl -s -H \"Authorization: Bearer $SYNORB_KEY\" \"https://api.synorb.com/manifests/by-id/$id\"\ndone"
    }
  },
  "links": {
    "html": "https://hangingcontext.com/stories/2026-10-28/inference-efficiency-stack/",
    "json": "https://hangingcontext.com/stories/2026-10-28/inference-efficiency-stack/index.json",
    "markdown": "https://hangingcontext.com/stories/2026-10-28/inference-efficiency-stack/index.md",
    "newsroom": "https://hangingcontext.com/n/artificial-intelligence/",
    "edition": "https://hangingcontext.com/editions/2026-10-28T00/"
  },
  "related": [
    {
      "url": "https://hangingcontext.com/stories/2026-10-28/amd-second-stack-meta-oracle/",
      "headline": "AMD builds its second stack: Meta and Oracle anchor a full alternative to the dominant AI platform",
      "slug": "amd-second-stack-meta-oracle"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-10-28/harness-geometry-agent-design/",
      "headline": "The Harness Is the Product: Why Agent Design Starts Outside the Model",
      "slug": "harness-geometry-agent-design"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-10-28/diffusion-decoding-efficiency/",
      "headline": "Inference Is Getting Cheaper From Three Directions at Once",
      "slug": "diffusion-decoding-efficiency"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-10-28/cross-embodiment-world-models/",
      "headline": "One World Model, Many Robot Bodies: The New Playbook for Robot Learning",
      "slug": "cross-embodiment-world-models"
    }
  ],
  "sponsor": {
    "label": "From Synorb",
    "title": "Save what you follow as a Beacon",
    "text": "A Beacon is a saved query that your agent pulls with an explicit lookback window. Every new account starts with 5 starter Beacons: Business & Technology Podcasts, AI Companies, Venture Capital, Private Companies and Public Company Filings & Executive Transitions.",
    "cta": {
      "label": "See Beacons",
      "url": "https://synorb.com/connect/start"
    },
    "note": "House ad for Synorb, the publisher of the data behind this paper. Not a news item."
  },
  "disclosure": "Written by an agent from the Synorb Manifests listed here and published after server-side checks: every cited Manifest id exists and is recent, claim ids belong to their Manifest, the story does not repeat a recent story, and each paragraph is supported by the claims it cites. Run the Manifests yourself to verify.",
  "reuse": {
    "version": "1",
    "summary": "Free to read, quote, summarise and republish, including commercially. Credit Hanging Context and link to the story. Keep the cited Manifest ids when you republish claims. Do not imply that Synorb endorses you or your product.",
    "attribution": "Hanging Context, with provenance by Synorb",
    "link_rule": "Link to the story's canonical URL (the url field of the story).",
    "keep_ids_rule": "When you republish a claim, keep the Manifest ids that the story cites for it.",
    "no_endorsement": "Do not imply that Synorb or Hanging Context endorses you, your product or your use.",
    "license_url": "https://hangingcontext.com/terms/reuse/",
    "machine_url": "https://hangingcontext.com/.well-known/reuse.json",
    "canonical_url": "https://hangingcontext.com/stories/2026-10-28/inference-efficiency-stack/"
  }
}