Your model's API is the attack surface

Learning what trained the model

Mindgard describes membership inference attacks, in which an adversary queries a model's API to determine whether specific data records were included in its training set. Membership inference against pretrained large language models barely outperforms random guessing, but prompted attacks against retrieval systems reached 0.89 AUC in black-box settings, and an attack reached 0.942 AUC against a fine-tuned Llama-3.2-3B-Instruct. That distinction is what makes the risk practical rather than theoretical: the most exposed layer is often the customized deployment, not the base model. m:…4966

The regulatory stakes follow from that. The European Data Protection Board's Opinion 28/2024 states a model is anonymous only if the likelihood of extracting personal data is insignificant, which turns an attack technique into a compliance test. The layers to test first are therefore the fine-tuned model and the retrieval pipeline, where the attacks demonstrably work. m:…4966

Copying the model through its answers

Model extraction works the same way, at larger scale. Mindgard explains that adversaries replicate model capabilities by querying APIs and training local student models on the outputs. Anthropic reported that three AI labs extracted over 16 million queries from Claude using approximately 24,000 fake accounts, and reasoning traces are considered the most valuable type of training data for frontier models. m:…8550

Restricting API outputs slows down model extraction but does not stop it, and distilled models are unlikely to retain the safety safeguards of the original model. Mindgard's Model Leeching attack cloned a ChatGPT-3.5-Turbo task for approximately $50 in API costs, so the damage is not only lost intellectual property: a cheap copy can be stripped of safety measures and used to build adversarial test benches. m:…8550

When data becomes instruction

A third risk needs no direct interaction with the model at all. Mindgard describes blind prompt injection, in which routine operational data, such as asset logs or physical labels, is interpreted as instructions by downstream AI systems. An attacker who can influence that data can influence AI behavior without ever touching the model. m:…9248

The recommended defenses are unglamorous but specific: understanding where information originates, knowing which databases, logs and business applications supply context to AI systems, and giving AI systems only the permissions necessary for their intended tasks. The point is that operational data flowing into AI systems should be treated as untrusted input, not as background fact. m:…9248

Testing that assumes failure

Offensive AI security, as Mindgard describes it, exists to find these problems before adversaries do. Unlike traditional software testing, it focuses on probabilistic failure modes in models, data pipelines and agents. 24 of 30 recent disclosures target AI agents or coding assistants, and Mindgard reported 156 vulnerabilities to 24 vendors, 98 of which were patched. m:…2119

Escape Tech makes a related argument about testing method. The company says relying on repeated AI pentest runs is inefficient and unreliable, because standard frontier models show significant variability across multiple attempts. Its multi-agent engine, Cascade Iguazu, found all nine seeded vulnerabilities in every execution of a benchmark test, while Claude Opus 5.5 found between six and eight per execution, averaging a 75% detection rate across four runs. Two independent executions with identical configurations can yield different results, and repeated runs lack a reliable stopping condition. That claim rests on the company's own testing, and buyers should verify it on their own systems, but the underlying point stands: a test that gives a different answer every run is not a test result. m:…8166

The defenses described across these items form a coherent program: restricting outputs to slow extraction, provenance and visibility for data flowing into AI systems, and constrained agent permissions to limit what a compromised agent can do, and offensive testing that assumes probabilistic failure rather than perfect refusal. No single defense closes the problem, but together they raise the cost of attack above the value of the copy. m:…8550 m:…9248 m:…2119