Security benchmarks commonly count every attack failure as an agent defensive success. The authors instead review tool traces and the legitimate task outcome and distinguish an injection ignored because the agent never acted on it, a genuine refusal, an unsafe side effect, and safe execution that fails the user. They describe major differences even when the…
Inspect the ClaimsCase · DiggingBeagle record
Prompt-injection benchmarks confuse silently ignored attacks with active refusals
A study audits 33,000 tool-using-agent trajectories and separates attacks refused, ignored and completed, revealing that a low attack success rate can conceal incomplete user tasks.
Scope
The reported percentages describe selected open models, datasets and prompting configurations, not the failure distribution for commercial agents. The label-blind evaluator itself introduces a judgment layer, and the same-weight 0.2 to 43 percent ASR example is a controlled system-prompt contrast rather than an in-the-wild attack-rate estimate.
Assessment profile
Assessment method- Setting
- evaluation
- Exploitation
- demonstrated
- Evidence
- Grade B
- Remediation
- not applicable
- AI role
- AGAINST AI
Basis and provenance
Enrich update: CLM-ENRICH-20261027-04 cites SRC-2026-CP21-ATOM-INJECTION-OUTCOME-CLASS at Abstract, same-weights system-prompt comparison and ATOM outcomes. New evidence concerns the bounded reported mechanism, prerequisite or outcome; setting evaluation; exploit status demonstrated; remediation status not_applicable. No independently comparable impact scale, affected production population or universal exploit success measure was established, so the impact band remains null. Original Claims remain individually attributable and any observed incident or simulated result is distinguished in the narrative.
Assessed Oct 27, 2026 using diggingbeagle.assessment/1.
Evidence basis: CLM-ENRICH-20261027-04 · Ignored, Not Refused: What Low Prompt-Injection Success Rates Hide in Small Tool-Using Agents
At a glance
A low attack-success number does not itself establish a safe useful agent when the agent may ignore the attacker only because it also ignores the user.
Read the implicationsThe reported percentages describe selected open models, datasets and prompting configurations, not the failure distribution for commercial agents. The label-blind evaluator itself introduces a judgment layer, and the same-weight 0.2 to 43 percent ASR example is a controlled system-prompt contrast rather than an in-the-wild attack-rate estimate.
Limits and uncertaintyFull account
Security benchmarks commonly count every attack failure as an agent defensive success. The authors instead review tool traces and the legitimate task outcome and distinguish an injection ignored because the agent never acted on it, a genuine refusal, an unsafe side effect, and safe execution that fails the user. They describe major differences even when the model weights do not change, because the agent system prompt and tools determine opportunities to cross a trust boundary.
Across 33,000 trajectories and eleven 2B-to-14B open models evaluated on InjecAgent and AgentDojo, the researchers classify 76 to 80 percent of failed attacks as ignored rather than explicitly refused and find 21.7 percent of AgentDojo runs preserved safety while failing to finish the original task. The reported percentages describe selected open models, datasets and prompting configurations, not the failure distribution for commercial agents. The label-blind evaluator itself introduces a judgment layer, and the same-weight 0.2 to 43 percent ASR example is a controlled system-prompt contrast rather than an in-the-wild attack-rate estimate.
Record refused, ignored, task-completed and unauthorized-tool-effect outcomes separately, retain execution traces and independently check side effects and legitimate task success. A low attack-success number does not itself establish a safe useful agent when the agent may ignore the attacker only because it also ignores the user. Primary evidence.
Additional evidence and trust boundary
ATOM makes the response matrix explicit: an attack can fail because it was never processed, because a model refused it, or because the legitimate task was abandoned. Those outcomes demand different remediations. Promoting ignored injections into a refusal-rate score can make the defense look robust even when it has not demonstrated boundary recognition. The 21.7 percent safe-but-incomplete figure also requires reporting utility separately. Supporting record.
Timeline
- Oct 16, 2026disclosure
Public disclosure
Claims & evidence
CLM-ENRICH-20261027-04With unchanged model weights, the authors report that altering the agent system prompt shifted InjecAgent attack success from 0.2 percent to 43 percent, showing that the evaluation harness materially changes measured risk.supported
Basis: reported finding
- supportsIgnored, Not Refused: What Low Prompt-Injection Success Rates Hide in Small Tool-Using Agentsprimary disclosure
Abstract, same-weights system-prompt comparison and ATOM outcomes
CLM-CP21-ATOM-INJECTION-OUTCOME-CLASS-1Across 33,000 trajectories and eleven 2B-to-14B open models evaluated on InjecAgent and AgentDojo, the researchers classify 76 to 80 percent of failed attacks as ignored rather than explicitly refused and find 21.7 percent of AgentDojo runs preserved safety while failing to finish the original task.supported
Basis: reported finding
- supportsIgnored, Not Refused: What Low Prompt-Injection Success Rates Hide in Small Tool-Using Agentsprimary disclosure
Abstract: 33,000 trajectories; eleven models; ignored/refused breakdown; ATOM safe-but-task-failed percentage
Implications
A low attack-success number does not itself establish a safe useful agent when the agent may ignore the attacker only because it also ignores the user.
Controls and mitigations
- Record refused, ignored, task-completed and unauthorized-tool-effect outcomes separately, retain execution traces and independently check side effects and legitimate task success.
Unknowns and contradictions
- The reported percentages describe selected open models, datasets and prompting configurations, not the failure distribution for commercial agents. The label-blind evaluator itself introduces a judgment layer, and the same-weight 0.2 to 43 percent ASR example is a controlled system-prompt contrast rather than an in-the-wild attack-rate estimate.
Sources and citation
Material revision history
- Oct 27, 2026 · Canonical change recorded · new in release · revision 196