Case · DiggingBeagle record

MetaPI shows low injection success can coincide with a defense that cannot finish tasks

MetaPI converts realistic agent utility tasks into adaptive prompt-injection evaluations, reusing AppWorld, VitaBench and Terminal-Bench environments. Its tests exposed cases where a defense appeared to block attack objectives while legitimate task completion also collapsed. This is a benchmark with controlled attacks and varying models. It does not establish a universal ranking of every production agent defense or that every requested harmful call was

Scope

The study differentiates whether the attacker can reach an injection surface, whether an unauthorized action is proposed and whether it is actually performed. A reported zero attack rate is not meaningful without the fraction of useful tasks the defender still completes.

Assessment profile

Assessment method
Setting
evaluation
Exploitation
demonstrated
Evidence
Grade B
Remediation
not applicable
AI role
AGAINST AI
Basis and provenance

evaluation setting with demonstrated status, based on the identified primary disclosure and its stated test or incident limits. The new Claim distinguishes source observations from hypothetical downstream effects. Impact band is unassigned because the scoped evidence does not justify a comparable quantitative score; no unobserved customer harm or loss is imputed. Remediation state: not_applicable.

Assessed Nov 13, 2026 using diggingbeagle.assessment/1.

At a glance

Mechanism and trust boundary

  1. 01

    The study differentiates whether the attacker can reach an injection surface, whether an unauthorized action is proposed and whether it is actually performed. A reported zero attack rate is not meaningful without the fraction of useful tasks the defender still completes.

  2. 02

    A guard that simply denies the workflow can have excellent attack scores but fail the purpose of the agent. Evaluate authorized completion and unauthorized effects against the same task distribution and tool access.

Claims & evidence

CLM-ENR243-019-DETAILMetaPI adapts AppWorld, VitaBench and Terminal-Bench utility tasks into indirect prompt-injection tests and reports that none of its evaluated defenses simultaneously maintains strong utility and security under its adaptive attack settings.supported

Basis: reported finding

Link to claim
CLM-R242-METAPI-DEFENSE-UTILITY-ATTRITION-LIMITSThis is a benchmark with controlled attacks and varying models. It does not establish a universal ranking of every production agent defense or that every requested harmful call was executed.supported

Basis: reported finding

Link to claim
CLM-R242-METAPI-DEFENSE-UTILITY-ATTRITION-MECHANISMMetaPI converts realistic agent utility tasks into adaptive prompt-injection evaluations, reusing AppWorld, VitaBench and Terminal-Bench environments. Its tests exposed cases where a defense appeared to block attack objectives while legitimate task completion also collapsed.supported

Basis: reported finding

Link to claim

Implications

A guard that simply denies the workflow can have excellent attack scores but fail the purpose of the agent. Evaluate authorized completion and unauthorized effects against the same task distribution and tool access.

Controls and mitigations

  • Audit both sides of each task: permitted end state and independently verified side effects, including rejected actions and tool availability.

Unknowns and contradictions

  • This is a benchmark with controlled attacks and varying models. It does not establish a universal ranking of every production agent defense or that every requested harmful call was executed.

Sources and citation

Material revision history

  1. Nov 13, 2026 · Canonical change recorded · new in release · revision 244