Agents fall for social engineering, and one clean retest proves nothing

Agents do not check before they act

A new benchmark for social engineering attacks on AI agents, with a five-dimensional evaluation framework across eight daily-life domains, finds that current frontier models are highly susceptible. The models tested showed proactive verification rates below 32% when subjected to social engineering attacks, meaning that in most cases the agent did not independently check a suspicious request before acting on it. That is the number that should make any team deploying a customer-facing agent pause: verification is the difference between an agent that checks and one that obeys. m:…6027

There is a constructive result in the same paper. The authors propose META-SEA, a co-evolutionary training framework, and report that it enables open-source models like Qwen3-8B to achieve defense performance comparable to closed-source models. That matters because it suggests the vulnerability is not a permanent property of open models: training that pairs attackers and defenders can close much of the gap. But the baseline result stands on its own. An agent that verifies less than a third of the time is an agent that will follow a plausible instruction from a stranger, and most production agents face strangers all day. m:…6027

The threat map is bigger than prompt injection

A systematization of the attack literature overturns a common assumption about where the risk sits. Tool, MCP and action-layer attacks account for the largest share of atomic attacks in agentic AI systems, while the prompt-injection and action-hijacking class has the broadest paper coverage. In other words, researchers write the most about prompt injection, but the attacks that actually exist in the largest numbers live at the tool and action layer, where an agent does things rather than says things. m:…6028

The same study finds that only 278 of 660 identified attack-safeguard relationships have been directly assessed in existing literature, so most proposed defenses have never been tested against the attacks they claim to stop. A separate benchmark for sparse-signal cybersecurity investigation, with 74 adversarial scenarios, shows what untested defenses look like in practice: current agent architectures exhibit high recall of 0.82 to 0.88 but low precision of 0.61 to 0.72 when operating in noisy environments. The agents find most of the real threats and also flag a large number of false alarms, and those failures are consistent across reasoning tiers and are not fixed by varying confounder density or event volume. m:…6028 m:…6036

Attackers now use agents too

The red-teaming side is also automating. A preprint on scaling adaptive prompt injection attacks with agent swarms shows that agentic attackers can reliably trick models into following instructions in tool output, even for models previously considered robust in the GraySwan Arena. Human red-teaming for prompt injection is nearing saturation for the Gemini model family, the authors say, which means human testers are finding fewer new failures while automated attackers keep finding them. The authors argue that static benchmarks are insufficient and that proactive, automated red-teaming is essential for future agent security. m:…6013

One methodological idea in that paper is worth noting: evaluating prompt injection robustness through non-malicious actions separates instruction-hierarchy adherence from general security knowledge, so a failure tells you the model followed the wrong instruction rather than that it lacks safety knowledge. Lumenova makes the same point from the testing side, arguing that red teaming for AI agents should include indirect attack paths like documents, emails and tool results, and that automated penetration testing of agents requires checking automated judge verdicts against human reviews. An agent's most dangerous input is often not the user's message but the document the user asked it to read. m:…6013 m:…4128

A clean retest is not proof

The most practical finding in this batch concerns how teams verify fixes. AI agents are non-deterministic, Lumenova argues, meaning an attack that fails once may succeed on a later try without any changes to the system. NIST researchers found that repeating the same attack 25 times increased the success rate from 57% to 80%, and the Cloud Security Alliance states that a single refusal does not prove resilience. Standard penetration testing relies on clean retests to verify security fixes, but for agents that approach is unreliable: a successful retest may simply mean the attack failed by chance. m:…4128

Lumenova's prescription is to document the specific rule or control that blocked the attack, because a rule on an action holds regardless of how the instruction is worded, and to require retesting after any change to the agent, its tools or its model. Untested attack categories should be reported as untested rather than passed. Centific's PAVE benchmark makes the same point for personal assistants: it tests task completion and restraint by checking action logs, device data and final replies, because transcript-based evaluations can fail to detect failed saves, dropped preferences and unauthorized actions. In one case an assistant deleted all three reminders after a hidden instruction in a note, and GPT-4o-mini's verdict differed from PAVE's on 7 of 20 tasks when viewing only the final reply. Taken together, the sources point to a testing standard that looks very different from current practice: repeated attacks, verified through rules rather than refusals, across the tool layer where most attacks actually live, with the honest admission that most safeguards have never been assessed at all. m:…4128 m:…3585