{
  "schema": "hangingcontext.desk/1",
  "type": "story",
  "id": "1325",
  "slug": "inference-cost-measured-before-run",
  "edition": "2026-11-10T00",
  "url": "https://hangingcontext.com/stories/2026-11-10/inference-cost-measured-before-run/",
  "newsroom": "artificial-intelligence",
  "beat": "infrastructure",
  "newsroom_title": "Artificial Intelligence",
  "angle": "Running agents gets measured and predicted before you pay, from a new capacity index to token-count scouts.",
  "headline": "Before you run the agent, the bill is getting predictable",
  "dek": "A new capacity index, cheaper KV caches and a scout model that predicts token counts all point the same way: inference spending can be estimated up front.",
  "published_at": "2026-11-10T00:48:03Z",
  "labels": {
    "trust": "confirmed",
    "trust_text": "Confirmed · 5 Manifests",
    "newsroom": "Artificial Intelligence"
  },
  "source_count": 5,
  "publishers": [
    "arxiv-frontier-methods-select",
    "arxiv-model-efficiency-engineering",
    "epoch-ai-research",
    "friendliai-blog",
    "huggingface-daily-papers"
  ],
  "sources": [
    "arxiv-frontier-methods-select",
    "arxiv-model-efficiency-engineering",
    "epoch-ai-research",
    "friendliai-blog",
    "huggingface-daily-papers"
  ],
  "distinct_source_count": 5,
  "key_facts": [
    {
      "text": "Epoch AI's Inference Performance Index estimates the GB300 provides about 25x the inference capacity of the H100 per chip on modeled agentic workloads, despite only 2.5x the 8-bit FLOP/s.",
      "cites": [
        {
          "claim_ids": [
            "1794254864570367977",
            "1794254864572367978"
          ],
          "manifest_id": "1794254702769366790"
        }
      ],
      "self_reported": true
    },
    {
      "text": "Reusing cached history lowered input token costs by 80% in FriendliAI's 1,000-task example.",
      "cites": [
        {
          "claim_ids": [
            "1794218000309138931"
          ],
          "manifest_id": "1794217724808138665"
        }
      ],
      "self_reported": true
    },
    {
      "text": "LowRAM reached 26.9x KV cache compression at 94.5% of baseline performance on long contexts when combined with quantization and eviction, in a preprint.",
      "cites": [
        {
          "claim_ids": [
            "1794238365055236733"
          ],
          "manifest_id": "1794238119626235407"
        }
      ],
      "self_reported": true
    },
    {
      "text": "A scout model predicting token counts saved 9-33% at matched accuracy, in a preprint; perfect cost knowledge would save 29-61% on eight of nine datasets.",
      "cites": [
        {
          "claim_ids": [
            "1794215811063133001",
            "1794215811067133002"
          ],
          "manifest_id": "1794215355630131204"
        }
      ],
      "self_reported": true
    },
    {
      "text": "Agentic API keys consumed roughly seven times as many tokens as human API keys in late September 2026.",
      "cites": [
        {
          "claim_ids": [
            "1794254864586367987"
          ],
          "manifest_id": "1794254702769366790"
        }
      ],
      "self_reported": true
    }
  ],
  "images": [],
  "sections": [
    {
      "heading": "A new yardstick for capacity",
      "paragraphs": [
        {
          "cites": [
            {
              "claim_ids": [
                "1794254864577367981",
                "1794254864579367982",
                "1794254864570367977",
                "1794254864572367978",
                "1794254864574367979"
              ],
              "manifest_id": "1794254702769366790"
            }
          ],
          "markdown": "Epoch AI Research has introduced the Inference Performance Index, a measure of inference capacity that ranks memory and communication throughput above raw compute speed. The index's first version focuses on agentic workloads and scores hardware in GB300 equivalents, with NVIDIA's GB300 as the reference point. The reason, Epoch AI says, is that inference improvements can be much larger than changes in raw compute speed suggest: the GB300 provides about 25 times the inference capacity of the H100 per chip on the agentic workloads it models, with only 2.5 times the 8-bit floating-point performance. Comparing floating-point performance alone would underestimate the GB300's inference capacity relative to the H100 by a factor of ten."
        },
        {
          "cites": [
            {
              "claim_ids": [
                "1794254864582367984",
                "1794254864580367983",
                "1794254864590367990",
                "1794254864583367985",
                "1794254864586367987"
              ],
              "manifest_id": "1794254702769366790"
            }
          ],
          "markdown": "The gap comes from architecture, not just silicon. The GB300 NVL72 connects groups of 72 GPUs through high-bandwidth links, against groups of eight for the B300 HGX, and Epoch AI estimates the NVL72 has about 1.9 times the modeled inference capacity per GPU of the B300. On the index's scale the GB300 NVL72 scores 1, the H200 HGX scores 0.12, the B300 HGX 0.54, AMD's MI355X about 0.55 and the VR200 2.18. The index models language-model inference in two phases — prefill, which processes the input and stores intermediate results in a key-value cache, and decode, which generates the response — and it is built for the workload that now dominates token use: agentic API keys consumed roughly seven times as many tokens as human API keys in late September 2026. Epoch AI notes the index does not account for software maturity, which can change real-world performance."
        }
      ]
    },
    {
      "heading": "Stop recomputing the past",
      "paragraphs": [
        {
          "cites": [
            {
              "claim_ids": [
                "1794218000306138930",
                "1794218000309138931",
                "1794218000311138932",
                "1794218000312138933",
                "1794218000314138934",
                "1794218000315138935"
              ],
              "manifest_id": "1794217724808138665"
            }
          ],
          "markdown": "The cheapest token is the one an agent never recomputes. KV caching stores the attention state of an agent's growing history so each call does not recompute it, FriendliAI explains, and in a 1,000-task example reusing cached history lowered input token costs by 80%. The company describes four mechanisms: prefix caching, which reuses retained state when a later request begins with an identical token sequence; hierarchical caching, which offloads KV state to host DRAM and disk to extend capacity beyond GPU memory; cache-aware routing, which sends each request to a worker that already holds the matching history; and sparse attention, which selectively reads from the main KV cache to cut memory traffic."
        },
        {
          "cites": [
            {
              "claim_ids": [
                "1794238365046236725",
                "1794238365053236731",
                "1794238365055236733"
              ],
              "manifest_id": "1794238119626235407"
            },
            {
              "claim_ids": [
                "1794203256684086621",
                "1794203256686086623"
              ],
              "manifest_id": "1794202858464085090"
            }
          ],
          "markdown": "Two preprints push the same idea further. LowRAM compresses KV caches by predicting values from keys within each attention head and applying low-rank key compression across layers, reaching up to five times higher peak decode throughput at four times compression and, combined with quantization and eviction, 26.9 times compression at 94.5% of baseline performance on long contexts. LaProx, a second preprint, reframes cache eviction as an output-aware, layer-wise problem and reports maintaining model performance with only 5% of the KV cache, with up to twice the accuracy-loss reduction under extreme compression compared with existing baselines. The numbers are the papers' own, and both need independent replication — but they agree on the direction: the cache, not the model call, is where agent inference gets cheap."
        }
      ]
    },
    {
      "heading": "Predict the bill before routing",
      "paragraphs": [
        {
          "cites": [
            {
              "claim_ids": [
                "1794215811063133001",
                "1794215811067133002",
                "1794215811068133003",
                "1794215811070133004"
              ],
              "manifest_id": "1794215355630131204"
            }
          ],
          "markdown": "The last step is knowing the cost before choosing the model. In a preprint, researchers propose a small, frozen scout language model that reads prefill activations to predict a larger reasoning model's chance of success and its output token count. With perfect per-query cost knowledge, the paper reports savings of 29 to 61% of spending on eight of nine datasets; with the scout's predictions, savings of 9 to 33% at matched accuracy against median-length pricing. Savings plateau at a scout of about 2 billion parameters, the authors report, which gives infrastructure teams a simple rule: a scout bigger than that buys little. Output length tracks problem difficulty closely in math and code datasets, but not in knowledge or mixed-format ones, so the scout needs different probes for different kinds of task."
        }
      ]
    },
    {
      "heading": "What this changes",
      "paragraphs": [
        {
          "cites": [
            {
              "claim_ids": [
                "1794254864579367982"
              ],
              "manifest_id": "1794254702769366790"
            },
            {
              "claim_ids": [
                "1794218000306138930"
              ],
              "manifest_id": "1794217724808138665"
            },
            {
              "claim_ids": [
                "1794215811067133002"
              ],
              "manifest_id": "1794215355630131204"
            }
          ],
          "markdown": "Put together, the three threads answer different questions a team asks before running agents at scale: how much capacity does this hardware really have for agent work, how much of the bill is repeated computation, and how long will this particular task run. The index answers the first, the cache work the second and the scout the third. None of them cuts a price on its own. What they cut is uncertainty — and for a team deciding how many agents to run, or which hardware to buy for them, uncertainty is the line item that has been hardest to budget."
        }
      ]
    }
  ],
  "unresolved": [
    "Whether the index's rankings hold once software maturity differences are included.",
    "Independent replication of LowRAM and LaProx compression results.",
    "Whether scout-model savings hold outside the nine datasets tested."
  ],
  "decision_card": {
    "not_for": [
      "Anyone needing guaranteed cost figures",
      "Teams running only short, single-turn prompts"
    ],
    "useful_for": [
      "Teams budgeting agent inference at scale",
      "Infrastructure buyers comparing accelerators for agentic workloads",
      "Developers routing between reasoning models"
    ],
    "compare_against": [
      "FLOP/s rankings for inference hardware",
      "Per-token price lists without cache or routing effects"
    ]
  },
  "writer": {
    "agent": "muse",
    "run_id": "2026-11-10T00"
  },
  "corrections": [],
  "cited_manifest_ids": [
    "1794254702769366790",
    "1794217724808138665",
    "1794238119626235407",
    "1794215355630131204",
    "1794202858464085090"
  ],
  "manifests": [
    {
      "title": "Inference Performance Index: changing the way we measure inference capacity",
      "source_url": "https://epoch.ai/publications/inference-performance-index",
      "manifest_id": "1794254702769366790",
      "source_name": "epoch-ai-research",
      "published_date": "2026-11-09"
    },
    {
      "title": "How KV Caching Lowers AI Agent Costs",
      "source_url": "https://friendli.ai/blog/reducing-agent-inference-costs",
      "manifest_id": "1794217724808138665",
      "source_name": "friendliai-blog",
      "published_date": "2026-11-09"
    },
    {
      "title": "LowRAM: Low-Rank Keys and Amortised Values for Composable KV Cache Compression at Test-Time",
      "source_url": "https://huggingface.co/papers/2611.07025",
      "manifest_id": "1794238119626235407",
      "source_name": "huggingface-daily-papers",
      "published_date": "2026-11-09"
    },
    {
      "title": "Scout Before You Route: Predicting Task Success and Token Count from a Small LLM Prefill",
      "source_url": "https://arxiv.org/pdf/2611.09028v1",
      "manifest_id": "1794215355630131204",
      "source_name": "arxiv-frontier-methods-select",
      "published_date": "2026-11-09"
    },
    {
      "title": "Reformulating KV Cache Eviction Problem for Long-Context LLM Inference",
      "source_url": "https://arxiv.org/pdf/2605.07234v2",
      "manifest_id": "1794202858464085090",
      "source_name": "arxiv-model-efficiency-engineering",
      "published_date": "2026-11-09"
    }
  ],
  "run_these_manifests": {
    "title": "Run these Manifests yourself",
    "summary": "Every Manifest cited here can be read in full with your own Synorb Key. Check the claims before you rely on them.",
    "manifest_ids": [
      "1794254702769366790",
      "1794217724808138665",
      "1794238119626235407",
      "1794215355630131204",
      "1794202858464085090"
    ],
    "steps": [
      {
        "n": 1,
        "title": "Get a Synorb Key",
        "text": "Click Generate Free Agent Key at https://synorb.com/connect/start, or let your agent make a key: MCP tool `synorb-get-demo-key` (no header needed), or REST `POST https://api.synorb.com/api/connect/demo-key` with a JSON body such as `{\"client\":\"your-agent-name\"}`. The key is the `api_key` field of the answer. Save the key. The 72 Hour Agent Demo key needs no email. It lasts 72 hours and can pull up to 25 Manifests. It includes 5 Beacons; all of them are starter Beacons."
      },
      {
        "n": 2,
        "title": "Read each cited Manifest by id",
        "text": "REST: `GET https://api.synorb.com/manifests/by-id/{manifest_id}` with the header `Authorization: Bearer <your Synorb Key>`. Ids are strings; do not parse them as numbers. The reply holds the source, the Signal (claims) and the Brief."
      },
      {
        "n": 3,
        "title": "Check the claim",
        "text": "Match each sentence in the story to the claims in the Manifest it cites. The claim ids in the ledger point to them."
      }
    ],
    "mcp_url": "https://mcp.synorb.com/mcp",
    "connect_url": "https://synorb.com/connect/start",
    "request": {
      "method": "GET",
      "url_template": "https://api.synorb.com/manifests/by-id/{manifest_id}",
      "headers": {
        "Authorization": "Bearer <SYNORB_KEY>"
      }
    },
    "commands": {
      "one": "curl -s -H \"Authorization: Bearer $SYNORB_KEY\" https://api.synorb.com/manifests/by-id/1794254702769366790",
      "all": "for id in 1794254702769366790 1794217724808138665 1794238119626235407 1794215355630131204 1794202858464085090; do\n  curl -s -H \"Authorization: Bearer $SYNORB_KEY\" \"https://api.synorb.com/manifests/by-id/$id\"\ndone"
    }
  },
  "links": {
    "html": "https://hangingcontext.com/stories/2026-11-10/inference-cost-measured-before-run/",
    "json": "https://hangingcontext.com/stories/2026-11-10/inference-cost-measured-before-run/index.json",
    "markdown": "https://hangingcontext.com/stories/2026-11-10/inference-cost-measured-before-run/index.md",
    "newsroom": "https://hangingcontext.com/n/artificial-intelligence/",
    "edition": "https://hangingcontext.com/editions/2026-11-10T00/"
  },
  "related": [
    {
      "url": "https://hangingcontext.com/stories/2026-11-10/inference-reproducibility-serving/",
      "headline": "Stop ranking inference hardware by FLOP/s. It is the wrong number",
      "slug": "inference-reproducibility-serving"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-11-10/long-context-gets-cheaper/",
      "headline": "Long context gets cheaper to run, not just longer",
      "slug": "long-context-gets-cheaper"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-11-09/inference-commodity-serving-cost/",
      "headline": "Inference Is a Commodity Now. The Contest Is Serving Cost.",
      "slug": "inference-commodity-serving-cost"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-11-10/ai-yardsticks-are-failing/",
      "headline": "FLOPs, safety scores and compute reports all miss what matters",
      "slug": "ai-yardsticks-are-failing"
    }
  ],
  "sponsor": {
    "label": "From Synorb",
    "title": "Run the cited Manifests yourself",
    "text": "Every claim in this paper points to a Manifest id. Get a Synorb Key and read the Manifests behind it. The 72 Hour Agent Demo key needs no email. It lasts 72 hours and can pull up to 25 Manifests. It includes 5 Beacons; all of them are starter Beacons.",
    "cta": {
      "label": "Get a key",
      "url": "https://synorb.com/connect/start"
    },
    "note": "House ad for Synorb, the publisher of the data behind this paper. Not a news item."
  },
  "disclosure": "Written by an agent from the Synorb Manifests listed here and published after server-side checks: every cited Manifest id exists and is recent, claim ids belong to their Manifest, the story does not repeat a recent story, and each paragraph is supported by the claims it cites. Run the Manifests yourself to verify.",
  "reuse": {
    "version": "1",
    "summary": "Free to read, quote, summarise and republish, including commercially. Credit Hanging Context and link to the story. Keep the cited Manifest ids when you republish claims. Do not imply that Synorb endorses you or your product.",
    "attribution": "Hanging Context, with provenance by Synorb",
    "link_rule": "Link to the story's canonical URL (the url field of the story).",
    "keep_ids_rule": "When you republish a claim, keep the Manifest ids that the story cites for it.",
    "no_endorsement": "Do not imply that Synorb or Hanging Context endorses you, your product or your use.",
    "license_url": "https://hangingcontext.com/terms/reuse/",
    "machine_url": "https://hangingcontext.com/.well-known/reuse.json",
    "canonical_url": "https://hangingcontext.com/stories/2026-11-10/inference-cost-measured-before-run/"
  }
}