Analysis · DiggingBeagle record
The injection failed. Was the agent still safe and useful?
New agent evaluations distinguish ignoring malicious text, actively refusing it, completing the legitimate task, and honoring runtime controls. Treating all four as one defense score hides distinct failures.
- Published
- Oct 27, 2026
Overview
An agent can ignore an injected command and still fail its user's task. It can also complete the task while violating a tool or approval rule. Three October studies make the same uncomfortable point from different angles: attack failure, user-task success and lawful execution are separate outcomes, not synonyms for secure behavior.
The missing dimensions
In ATOM, Farhan Zarif and colleagues examined roughly 33,000 trajectories involving eleven open models and two familiar agent-attack benchmarks. Among attacks that failed to achieve their objective, around 76 to 80 percent were simply ignored and only about 17 to 18 percent triggered active refusal. The reported AgentDojo evaluation also found that roughly 21.7 percent of runs avoided the malicious instruction but failed the genuine user request. Those are conditional measurements in particular benchmarks, not field rates.
TraceTrial-Bench, led by Zhiyong Cao, evaluated 3,420 trajectories across 95 executable tasks, nine models and four harnesses. It reported that 31.1 percent of task-successful trajectories still broke at least one harness correctness criterion. Those seven criteria include governance and verification, not merely text-answer correctness.
| Observation | What it measures | What it does not prove |
|---|---|---|
| Attack ignored in ATOM | The attacker did not get the requested deviation | The agent actively recognized or refused the attack |
| Legitimate task failed in ATOM | User utility lost despite attack failure | Successful attacker control |
| Task successful but harness-invalid in TraceTrial | Output reached a task goal while another execution rule failed | That each violation caused unauthorized data access |
| Targeted deviation in NTBench | An attacker can distort one near-correct detail | That the whole task visibly failed |
Why near-correct answers need a separate test
Yuyang Gong and colleagues describe injections that keep much of the requested task intact while modifying a critical result. Their reported gains over fixed-target attacks were 9.3 to 35.8 percentage points in incorrect-answer settings and 1.7 to 17.3 points in harmful-action settings. These experiments do not imply equivalent harm in production, but they illustrate why testing only catastrophic task derailment misses subtler manipulation.
The relevant control is not a prettier refusal message. Each action should carry an explicit authorized objective, input provenance, scoped permission, and observed effect receipt. The harness should then judge both what the user got and how the agent got it. A success label should not erase a wrong recipient, unauthorized side effect or unverified result.
The ATOM paper, TraceTrial study and NTBench experiments share enough concern to compare but not a common dataset or denominator. No single pooled rate follows from them.
Practical test: track task_completed, attack_goal_achieved, independent_policy_passed and effect_verified independently. The interesting cases are precisely those in which their answers disagree.