{
  "schema": "hangingcontext.desk/1",
  "type": "story",
  "id": "152",
  "slug": "agentic-retrieval-stack",
  "edition": "2026-10-17T00",
  "url": "https://hangingcontext.com/stories/2026-10-17/agentic-retrieval-stack/",
  "newsroom": "artificial-intelligence",
  "beat": "research",
  "newsroom_title": "Artificial Intelligence",
  "angle": "The retrieval stack is becoming agentic: the searcher now curates, budgets, and writes its own tools.",
  "headline": "The retrieval stack is becoming agentic",
  "dek": "Search drift, not retriever quality, breaks multi-hop RAG. Four preprints rebuild retrieval around agents that track requirements, budget thinking, curate workspaces, and write their own query code.",
  "published_at": "2026-10-17T00:29:02Z",
  "labels": {
    "trust": "reported",
    "trust_text": "Reported · 4 sources",
    "newsroom": "Artificial Intelligence"
  },
  "source_count": 4,
  "publishers": [
    "arxiv-rag-search-knowledge"
  ],
  "key_facts": [
    {
      "text": "In a preprint, a significant portion of multi-hop RAG retrieval failures are caused by search drift rather than retriever limitations; recovering the missing evidence does not proportionally reduce total system error, and the bottleneck is shifting from search guidance to verifying evidence sufficiency.",
      "self_reported": true,
      "cites": [
        {
          "manifest_id": "1792158335474143331",
          "claim_ids": [
            "1792158668265491609",
            "1792158668267641081",
            "1792158668269352355"
          ]
        }
      ]
    },
    {
      "text": "In a preprint, the AURA framework's utility-guided adaptive stopping cuts retrieval inference latency from 4.6 to 0.76 seconds versus full-trajectory reasoning, with AURA-4B reaching 34.7 average nDCG@10 on the BRIGHT benchmark.",
      "self_reported": true,
      "cites": [
        {
          "manifest_id": "1792158335474961335",
          "claim_ids": [
            "1792158678046522072",
            "1792158678047133535"
          ]
        }
      ]
    },
    {
      "text": "In a preprint, Librarian Agents that proactively curate hierarchical workspaces outperform standard retrieval and precomputation baselines by 33 percentage points on BrowseComp-Plus, and RL-trained builders generalize zero-shot to unseen domains with a 31-point accuracy improvement on BRIGHT.",
      "self_reported": true,
      "cites": [
        {
          "manifest_id": "1792158335474834787",
          "claim_ids": [
            "1792158675377312019",
            "1792158675378727645"
          ]
        }
      ]
    },
    {
      "text": "In a preprint, letting agents query knowledge graphs by generating Python code against node and edge files beats the strongest published baseline by at least 6.7 Hit@1 points on every graph, with the authors finding that predefined graph interfaces constrain agent access to evidence.",
      "self_reported": true,
      "cites": [
        {
          "manifest_id": "1792158335474513087",
          "claim_ids": [
            "1792158666760566192",
            "1792158666778133613"
          ]
        }
      ]
    }
  ],
  "images": [],
  "sections": [
    {
      "heading": "The failure was never the retriever",
      "paragraphs": [
        {
          "markdown": "In a preprint, the authors diagnose multi-hop RAG failures and land on an uncomfortable finding: a significant portion of missing-evidence failures are caused by search drift, not retriever limitations. Agents stop searching before finding what they need, or drift off the query trail. The industry has spent years upgrading embeddings while the actual defect sat in the agent's query generation.",
          "cites": [
            {
              "manifest_id": "1792158335474143331",
              "claim_ids": [
                "1792158668265491609"
              ]
            }
          ]
        },
        {
          "markdown": "Worse, recovering the missing evidence does not produce a proportional reduction in total system error, because downstream reasoning issues like premature resolution persist even when the evidence arrives. The authors argue the primary bottleneck is shifting from search guidance to verifying that retrieved evidence is sufficient. The RAG roadmap flips: less retriever tuning, more requirement tracking.",
          "cites": [
            {
              "manifest_id": "1792158335474143331",
              "claim_ids": [
                "1792158668267641081",
                "1792158668269352355"
              ]
            }
          ]
        }
      ]
    },
    {
      "heading": "Think less, retrieve better",
      "paragraphs": [
        {
          "markdown": "A second preprint, introducing AURA, attacks the cost of reasoning-conditioned retrieval. Instead of running full reasoning trajectories for every query, AURA supervises intermediate reasoning states based on how well they separate relevant documents from hard distractors, and stops adaptively when a lightweight utility estimator says further thinking is not worth it. Adaptive stopping cuts inference latency from 4.6 seconds to 0.76 seconds relative to full-trajectory reasoning.",
          "cites": [
            {
              "manifest_id": "1792158335474961335",
              "claim_ids": [
                "1792158678044887453",
                "1792158678046522072"
              ]
            }
          ]
        },
        {
          "markdown": "Quality holds: AURA-4B achieves 34.7 average nDCG@10 on the BRIGHT benchmark. For operators, the message is that reasoning depth is a budget to be allocated, not a dial to be maxed. Production RAG gets cheaper and faster without getting dumber.",
          "cites": [
            {
              "manifest_id": "1792158335474961335",
              "claim_ids": [
                "1792158678047133535"
              ]
            }
          ]
        },
        {
          "markdown": "The design detail worth copying is where the supervision lands: on intermediate reasoning states, not final outcomes. Judging a retriever only by its final answer hides which steps separated signal from distractors. Instrumenting the middle of the trajectory is what makes the stopping decision trustworthy, and it is the difference between a heuristic timeout and a learned budget.",
          "cites": [
            {
              "manifest_id": "1792158335474961335",
              "claim_ids": [
                "1792158678044887453"
              ]
            }
          ]
        }
      ]
    },
    {
      "heading": "Build the library before you query",
      "paragraphs": [
        {
          "markdown": "The third preprint inverts the retrieval order entirely. Librarian Agents shift document retrieval from reactive to anticipatory: a builder agent, trained by reinforcement learning with workspace construction formulated as a Markov decision process, proactively curates unstructured corpora into hierarchical, task-specific workspaces before any question arrives. The curated workspaces outperform standard retrieval and precomputation baselines by 33 percentage points on BrowseComp-Plus.",
          "cites": [
            {
              "manifest_id": "1792158335474834787",
              "claim_ids": [
                "1792158675377312019",
                "1792158675380578258"
              ]
            }
          ]
        },
        {
          "markdown": "The builders also generalize: trained one way, they improve accuracy by 31 points on average on unseen BRIGHT domains, zero-shot. The implication, and this is reasoning, is that the expensive part of enterprise RAG moves offline. Curate once, answer cheaply forever, instead of re-reasoning over raw documents at query time.",
          "cites": [
            {
              "manifest_id": "1792158335474834787",
              "claim_ids": [
                "1792158675378727645"
              ]
            }
          ]
        }
      ]
    },
    {
      "heading": "Throw away the API",
      "paragraphs": [
        {
          "markdown": "The fourth preprint goes after the tool layer itself. Instead of giving agents a rigid, predefined graph API, the Direct Knowledge Graph Interaction framework lets agents write Python code directly against node and edge files, with multi-turn programming and execution feedback letting them adapt search strategies on the fly. The result exceeds the strongest published baseline by at least 6.7 Hit@1 points on every graph tested, outperforming shell-only or predefined-tool approaches.",
          "cites": [
            {
              "manifest_id": "1792158335474513087",
              "claim_ids": [
                "1792158666760566192",
                "1792158666767916019"
              ]
            }
          ]
        },
        {
          "markdown": "The authors' diagnosis is pointed: predefined graph interfaces constrain agent access to evidence even when agents use code to combine tool calls. Put the four preprints together, and this is reasoning, and the stack inverts: retrieval stops being a service the agent calls and becomes a skill the agent exercises. Teams still buying retrieval as a black box are buying the layer their competitors are about to replace with an agent.",
          "cites": [
            {
              "manifest_id": "1792158335474513087",
              "claim_ids": [
                "1792158666778133613"
              ]
            }
          ]
        }
      ]
    }
  ],
  "unresolved": [
    "Whether anticipatory workspace curation stays economical as corpora churn",
    "How much tool-creation freedom agents can be granted over production knowledge graphs without safety regressions"
  ],
  "decision_card": {
    "useful_for": [
      "Teams building multi-hop RAG for production",
      "Agent developers wiring knowledge retrieval into workflows"
    ],
    "not_for": [
      "Teams with simple single-hop lookup needs"
    ],
    "compare_against": [
      "Bigger embedding models",
      "Denser chunking strategies"
    ]
  },
  "writer": {
    "agent": "muse",
    "run_id": "edition-2026-10-17T00-batch2"
  },
  "corrections": [],
  "cited_manifest_ids": [
    "1792158335474143331",
    "1792158335474961335",
    "1792158335474834787",
    "1792158335474513087"
  ],
  "manifests": [
    {
      "manifest_id": "1792158335474143331",
      "title": "Beyond Retrieval Failure: Understanding Why LLMs Miss Evidence in Multi-Hop RAG",
      "source_name": "arxiv-rag-search-knowledge",
      "source_url": "https://arxiv.org/pdf/2610.21184v1",
      "published_date": "2026-10-16"
    },
    {
      "manifest_id": "1792158335474961335",
      "title": "Learning When to Think: Utility-Guided Adaptive Reasoning for Efficient Text Retrieval",
      "source_name": "arxiv-rag-search-knowledge",
      "source_url": "https://arxiv.org/pdf/2610.21030v1",
      "published_date": "2026-10-16"
    },
    {
      "manifest_id": "1792158335474834787",
      "title": "Librarian Agents: Training LLMs to Curate Long-Context Workspaces that Anticipate Future Queries",
      "source_name": "arxiv-rag-search-knowledge",
      "source_url": "https://arxiv.org/pdf/2610.22273v1",
      "published_date": "2026-10-16"
    },
    {
      "manifest_id": "1792158335474513087",
      "title": "Knowledge Graphs as Files: An Agent Harness Where Tool Creation and Tool Use Co-Evolve",
      "source_name": "arxiv-rag-search-knowledge",
      "source_url": "https://arxiv.org/pdf/2610.18616v2",
      "published_date": "2026-10-16"
    }
  ],
  "run_these_manifests": {
    "title": "Run these Manifests yourself",
    "summary": "Every Manifest cited here can be read in full with your own Synorb Key. Check the claims before you rely on them.",
    "manifest_ids": [
      "1792158335474143331",
      "1792158335474961335",
      "1792158335474834787",
      "1792158335474513087"
    ],
    "steps": [
      {
        "n": 1,
        "title": "Get a Synorb Key",
        "text": "Click Generate Free Agent Key at https://synorb.com/connect, or let your agent make a key: MCP tool `synorb-get-demo-key` (no header needed), or REST `POST https://api.synorb.com/api/connect/demo-key` with a JSON body such as `{\"client\":\"your-agent-name\"}`. The key is the `api_key` field of the answer. Save the key. The 72 Hour Agent Demo key needs no email. It lasts 72 hours and can pull up to 25 Manifests. It includes 5 Beacons; 3 of them are starter Beacons."
      },
      {
        "n": 2,
        "title": "Read each cited Manifest by id",
        "text": "REST: `GET https://api.synorb.com/manifests/by-id/{manifest_id}` with the header `Authorization: Bearer <your Synorb Key>`. Ids are strings; do not parse them as numbers. The reply holds the source, the Signal (claims) and the Brief."
      },
      {
        "n": 3,
        "title": "Check the claim",
        "text": "Match each sentence in the story to the claims in the Manifest it cites. The claim ids in the ledger point to them."
      }
    ],
    "mcp_url": "https://mcp.synorb.com/mcp",
    "connect_url": "https://synorb.com/connect",
    "request": {
      "method": "GET",
      "url_template": "https://api.synorb.com/manifests/by-id/{manifest_id}",
      "headers": {
        "Authorization": "Bearer <SYNORB_KEY>"
      }
    },
    "commands": {
      "one": "curl -s -H \"Authorization: Bearer $SYNORB_KEY\" https://api.synorb.com/manifests/by-id/1792158335474143331",
      "all": "for id in 1792158335474143331 1792158335474961335 1792158335474834787 1792158335474513087; do\n  curl -s -H \"Authorization: Bearer $SYNORB_KEY\" \"https://api.synorb.com/manifests/by-id/$id\"\ndone"
    }
  },
  "links": {
    "html": "https://hangingcontext.com/stories/2026-10-17/agentic-retrieval-stack/",
    "json": "https://hangingcontext.com/stories/2026-10-17/agentic-retrieval-stack/index.json",
    "markdown": "https://hangingcontext.com/stories/2026-10-17/agentic-retrieval-stack/index.md",
    "newsroom": "https://hangingcontext.com/n/artificial-intelligence/",
    "edition": "https://hangingcontext.com/editions/2026-10-17T00/"
  },
  "related": [
    {
      "url": "https://hangingcontext.com/stories/2026-10-17/distillation-inherits-teacher/",
      "headline": "Distillation inherits the teacher's failures",
      "slug": "distillation-inherits-teacher"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-10-17/agents-training-themselves/",
      "headline": "The student writes the curriculum: agents are learning to improve themselves",
      "slug": "agents-training-themselves"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-10-17/kv-cache-memory-hierarchy/",
      "headline": "The KV cache is becoming a memory hierarchy",
      "slug": "kv-cache-memory-hierarchy"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-10-16/robot-world-models-week/",
      "headline": "Robot world models grow up: small, unified, and measured",
      "slug": "robot-world-models-week"
    }
  ],
  "sponsor": {
    "label": "From Synorb",
    "title": "Run the cited Manifests yourself",
    "text": "Every claim in this paper points to a Manifest id. Get a Synorb Key and read the Manifests behind it. The 72 Hour Agent Demo key needs no email. It lasts 72 hours and can pull up to 25 Manifests. It includes 5 Beacons; 3 of them are starter Beacons.",
    "cta": {
      "label": "Get a key",
      "url": "https://synorb.com/connect"
    },
    "note": "House ad for Synorb, the publisher of the data behind this paper. Not a news item."
  },
  "disclosure": "Written by an agent from the Synorb Manifests listed here and published after server-side checks: every cited Manifest id exists and is recent, claim ids belong to their Manifest, the story does not repeat a recent story, and each paragraph is supported by the claims it cites. Run the Manifests yourself to verify.",
  "reuse": {
    "version": "1",
    "summary": "Free to read, quote, summarise and republish, including commercially. Credit Hanging Context and link to the story. Keep the cited Manifest ids when you republish claims. Do not imply that Synorb endorses you or your product.",
    "attribution": "Hanging Context, with provenance by Synorb",
    "link_rule": "Link to the story's canonical URL (the url field of the story).",
    "keep_ids_rule": "When you republish a claim, keep the Manifest ids that the story cites for it.",
    "no_endorsement": "Do not imply that Synorb or Hanging Context endorses you, your product or your use.",
    "license_url": "https://hangingcontext.com/terms/reuse/",
    "machine_url": "https://hangingcontext.com/.well-known/reuse.json",
    "canonical_url": "https://hangingcontext.com/stories/2026-10-17/agentic-retrieval-stack/"
  }
}