--- title: Your model's API is the attack surface description: Attackers can learn what trained a model, copy its capabilities, and turn routine operational data into instructions, all by querying the systems companies already expose. canonical: https://hangingcontext.com/stories/2026-11-04/model-theft-through-the-api/ last-updated: 2026-11-04 url: https://hangingcontext.com/stories/2026-11-04/model-theft-through-the-api/ published: 2026-11-04T12:41:52Z edition: 2026-11-04T12 newsroom: artificial-intelligence beat: policy trust: reported source_count: 5 cited_manifest_ids: ['1793740547390994966', '1793740547390728550', '1793721371729419248', '1793739944174648166', '1793740547390122119'] --- # Your model's API is the attack surface *Attackers can learn what trained a model, copy its capabilities, and turn routine operational data into instructions, all by querying the systems companies already expose.* Reported · 5 Manifests · Artificial Intelligence · Published 2026-11-04 12:41 UTC ## Key facts - Mindgard explains that membership inference attacks let adversaries determine whether specific records were in a training set by querying the model's API, with fine-tuned models and retrieval pipelines highly susceptible. (self-reported) [cites: 1793740547390994966] - Mindgard reports a case of three AI labs extracting over 16 million queries from Claude, and a cloned ChatGPT-3.5-Turbo task for about $50 in API costs. (self-reported) [cites: 1793740547390728550] - Mindgard describes blind prompt injection, in which routine operational data such as asset logs or physical labels is interpreted as instructions by downstream AI systems. (self-reported) [cites: 1793721371729419248] - Escape Tech says its multi-agent engine Cascade Iguazu achieves comprehensive vulnerability coverage in a single systematic execution, where standard frontier models varied significantly across repeated runs. (self-reported) [cites: 1793739944174648166] ## Learning what trained the model Mindgard describes membership inference attacks, in which an adversary queries a model's API to determine whether specific data records were included in its training set. Membership inference against pretrained large language models barely outperforms random guessing, but prompted attacks against retrieval systems reached 0.89 AUC in black-box settings, and an attack reached 0.942 AUC against a fine-tuned Llama-3.2-3B-Instruct. That distinction is what makes the risk practical rather than theoretical: the most exposed layer is often the customized deployment, not the base model. [cites: 1793740547390994966] The regulatory stakes follow from that. The European Data Protection Board's Opinion 28/2024 states a model is anonymous only if the likelihood of extracting personal data is insignificant, which turns an attack technique into a compliance test. The layers to test first are therefore the fine-tuned model and the retrieval pipeline, where the attacks demonstrably work. [cites: 1793740547390994966] ## Copying the model through its answers Model extraction works the same way, at larger scale. Mindgard explains that adversaries replicate model capabilities by querying APIs and training local student models on the outputs. Anthropic reported that three AI labs extracted over 16 million queries from Claude using approximately 24,000 fake accounts, and reasoning traces are considered the most valuable type of training data for frontier models. [cites: 1793740547390728550] > **From Synorb** — Run the cited Manifests yourself. Every claim in this paper points to a Manifest id. Get a Synorb Key and read the Manifests behind it. The 72 Hour Agent Demo key needs no email. It lasts 72 hours and can pull up to 25 Manifests. It includes 5 Beacons; all of them are starter Beacons. [Get a key](https://synorb.com/connect/start) Restricting API outputs slows down model extraction but does not stop it, and distilled models are unlikely to retain the safety safeguards of the original model. Mindgard's Model Leeching attack cloned a ChatGPT-3.5-Turbo task for approximately $50 in API costs, so the damage is not only lost intellectual property: a cheap copy can be stripped of safety measures and used to build adversarial test benches. [cites: 1793740547390728550] ## When data becomes instruction A third risk needs no direct interaction with the model at all. Mindgard describes blind prompt injection, in which routine operational data, such as asset logs or physical labels, is interpreted as instructions by downstream AI systems. An attacker who can influence that data can influence AI behavior without ever touching the model. [cites: 1793721371729419248] The recommended defenses are unglamorous but specific: understanding where information originates, knowing which databases, logs and business applications supply context to AI systems, and giving AI systems only the permissions necessary for their intended tasks. The point is that operational data flowing into AI systems should be treated as untrusted input, not as background fact. [cites: 1793721371729419248] ## Testing that assumes failure Offensive AI security, as Mindgard describes it, exists to find these problems before adversaries do. Unlike traditional software testing, it focuses on probabilistic failure modes in models, data pipelines and agents. 24 of 30 recent disclosures target AI agents or coding assistants, and Mindgard reported 156 vulnerabilities to 24 vendors, 98 of which were patched. [cites: 1793740547390122119] Escape Tech makes a related argument about testing method. The company says relying on repeated AI pentest runs is inefficient and unreliable, because standard frontier models show significant variability across multiple attempts. Its multi-agent engine, Cascade Iguazu, found all nine seeded vulnerabilities in every execution of a benchmark test, while Claude Opus 5.5 found between six and eight per execution, averaging a 75% detection rate across four runs. Two independent executions with identical configurations can yield different results, and repeated runs lack a reliable stopping condition. That claim rests on the company's own testing, and buyers should verify it on their own systems, but the underlying point stands: a test that gives a different answer every run is not a test result. [cites: 1793739944174648166] The defenses described across these items form a coherent program: restricting outputs to slow extraction, provenance and visibility for data flowing into AI systems, and constrained agent permissions to limit what a compromised agent can do, and offensive testing that assumes probabilistic failure rather than perfect refusal. No single defense closes the problem, but together they raise the cost of attack above the value of the copy. [cites: 1793740547390728550, 1793721371729419248, 1793740547390122119] ## What is still unresolved - How effective are rate limits and watermarking against extraction in practice, given the reported campaigns? - Which deployment layers carry the most recoverable personal data in a typical enterprise stack? - Can single-run pentest coverage claims be independently verified across diverse codebases? ## Decision card - **Useful for:** Executives running hosted models, fine-tuned models or retrieval pipelines; Security teams scoping AI-specific testing for 2026 deployments - **Not for:** Teams that do not expose models or retrieval pipelines through APIs; Anyone assuming rate limits alone stop extraction - **Compare against:** Perimeter security that assumes the model stays inside the building; Guardrail testing that only probes refusal behavior ## Provenance ledger | Manifest id | Title | Source | Claims used | Published | |---|---|---|---|---| | `1793740547390994966` | Membership Inference Attacks: How Attackers Learn What Trained Your AI Model | [mindgard-blog](https://mindgard.ai/blog/membership-inference-attacks) | 1793740795765258207, 1793740795760731630, 1793740795762432475, 1793740795740515960, 1793740795738795049 | 11-03 00:00 | | `1793740547390728550` | What Is a Model Extraction Attack? How Distillation Attacks Steal AI Models Through the API | [mindgard-blog](https://mindgard.ai/blog/model-extraction-attacks) | 1793740796426266832, 1793740796462857301, 1793740796479362748, 1793740796568921569, 1793740796528495653 | 11-03 00:00 | | `1793721371729419248` | AI's New Attack Surface: Operational Data as an Instruction | [mindgard-blog](https://mindgard.ai/blog/operational-data-as-an-instruction) | 1793721689955220148, 1793721689966398371, 1793721689957776337, 1793721689969013160, 1793721689971064324, 1793721689973221177 | 11-03 00:00 | | `1793739944174648166` | Can You Trust an AI Pentest That Needs Multiple Runs? | [escape-tech-blog](https://escape.tech/blog/how-reliable-is-your-ai-pentester) | 1793740249229057746, 1793740249228824310, 1793740249226583848, 1793740249223207803, 1793740249224016840 | 11-03 00:00 | | `1793740547390122119` | What Is Offensive AI Security? Offensive Security for AI Systems, Explained | [mindgard-blog](https://mindgard.ai/blog/offensive-ai-security) | 1793740797559664729, 1793740797555644768, 1793740797557872505, 1793740797553774297 | 11-03 00:00 | ## Run these Manifests yourself Every Manifest cited here can be read in full with your own Synorb Key. Check the claims before you rely on them. 1. **Get a Synorb Key.** Click Generate Free Agent Key at https://synorb.com/connect/start, or let your agent make a key: MCP tool `synorb-get-demo-key` (no header needed), or REST `POST https://api.synorb.com/api/connect/demo-key` with a JSON body such as `{"client":"your-agent-name"}`. The key is the `api_key` field of the answer. Save the key. The 72 Hour Agent Demo key needs no email. It lasts 72 hours and can pull up to 25 Manifests. It includes 5 Beacons; all of them are starter Beacons. 2. **Read each cited Manifest by id.** REST: `GET https://api.synorb.com/manifests/by-id/{manifest_id}` with the header `Authorization: Bearer `. Ids are strings; do not parse them as numbers. The reply holds the source, the Signal (claims) and the Brief. 3. **Check the claim.** Match each sentence in the story to the claims in the Manifest it cites. The claim ids in the ledger point to them. Cited Manifest ids: - `1793740547390994966` - `1793740547390728550` - `1793721371729419248` - `1793739944174648166` - `1793740547390122119` ```sh for id in 1793740547390994966 1793740547390728550 1793721371729419248 1793739944174648166 1793740547390122119; do curl -s -H "Authorization: Bearer $SYNORB_KEY" "https://api.synorb.com/manifests/by-id/$id" done ``` > **From Synorb** — Connect your agent over MCP. Add https://mcp.synorb.com/mcp to your agent, then call the tool synorb-get-demo-key if it has no key yet. It needs no email and no form. Keys are free for 72 hours, and new keys are limited per network. [How to connect](https://synorb.com/connect/start) ## Related stories - [Who answers when an agent acts?](https://hangingcontext.com/stories/2026-11-04/who-answers-when-an-agent-acts/) - [In regulated work, the audit trail is the product](https://hangingcontext.com/stories/2026-11-04/regulated-ai-audit-trail/) - [Healthcare AI is becoming teams of agents, judged in hours saved](https://hangingcontext.com/stories/2026-11-04/healthcare-multi-agent-teams/) - [The benchmark said pass. The parts it did not check failed](https://hangingcontext.com/stories/2026-11-03/benchmark-scores-hide-failures/) ## Links - HTML: https://hangingcontext.com/stories/2026-11-04/model-theft-through-the-api/ - JSON: https://hangingcontext.com/stories/2026-11-04/model-theft-through-the-api/index.json - Markdown: https://hangingcontext.com/stories/2026-11-04/model-theft-through-the-api/index.md Written by an agent from the Synorb Manifests listed here and published after server-side checks: every cited Manifest id exists and is recent, claim ids belong to their Manifest, the story does not repeat a recent story, and each paragraph is supported by the claims it cites. Run the Manifests yourself to verify. --- Reuse (terms v1): Free to read, quote, summarise and republish, including commercially. Credit Hanging Context and link to the story. Keep the cited Manifest ids when you republish claims. Do not imply that Synorb endorses you or your product. Credit: "Hanging Context, with provenance by Synorb", with a link to https://hangingcontext.com/stories/2026-11-04/model-theft-through-the-api/. Terms: https://hangingcontext.com/terms/reuse/