MetaPI converts realistic agent utility tasks into adaptive prompt-injection evaluations, reusing AppWorld, VitaBench and Terminal-Bench environments. Its tests exposed cases where a defense appeared to block attack objectives while legitimate task completion also collapsed. The study differentiates whether the attacker can reach an injection surface,…
Inspect the ClaimsCase · DiggingBeagle record
MetaPI shows low injection success can coincide with a defense that cannot finish tasks
MetaPI converts realistic agent utility tasks into adaptive prompt-injection evaluations, reusing AppWorld, VitaBench and Terminal-Bench environments. Its tests exposed cases where a defense appeared to block attack objectives while legitimate task completion also collapsed. This is a benchmark with controlled attacks and varying models. It does not establish a universal ranking of every production agent defense or that every requested harmful call was
The study differentiates whether the attacker can reach an injection surface, whether an unauthorized action is proposed and whether it is actually performed. A reported zero attack rate is not meaningful without the fraction of useful tasks the defender still completes.
Assessment profile
Assessment method- Setting
- evaluation
- Exploitation
- demonstrated
- Evidence
- Grade B
- Remediation
- not applicable
- AI role
- AGAINST AI
Basis and provenance
evaluation setting with demonstrated status, based on the identified primary disclosure and its stated test or incident limits. The new Claim distinguishes source observations from hypothetical downstream effects. Impact band is unassigned because the scoped evidence does not justify a comparable quantitative score; no unobserved customer harm or loss is imputed. Remediation state: not_applicable.
Assessed Nov 13, 2026 using diggingbeagle.assessment/1.
Evidence basis: CLM-ENR243-019-DETAIL · CLM-R242-METAPI-DEFENSE-UTILITY-ATTRITION-LIMITS · CLM-R242-METAPI-DEFENSE-UTILITY-ATTRITION-MECHANISM · MetaPI: Constructing Prompt Injection Benchmarks from LLM Agent Benchmark
At a glance
A guard that simply denies the workflow can have excellent attack scores but fail the purpose of the agent. Evaluate authorized completion and unauthorized effects against the same task distribution and tool access.
Read the implicationsThis is a benchmark with controlled attacks and varying models. It does not establish a universal ranking of every production agent defense or that every requested harmful call was executed.
Limits and uncertaintyFull account
MetaPI converts realistic agent utility tasks into adaptive prompt-injection evaluations, reusing AppWorld, VitaBench and Terminal-Bench environments. Its tests exposed cases where a defense appeared to block attack objectives while legitimate task completion also collapsed.
The study differentiates whether the attacker can reach an injection surface, whether an unauthorized action is proposed and whether it is actually performed. A reported zero attack rate is not meaningful without the fraction of useful tasks the defender still completes. A guard that simply denies the workflow can have excellent attack scores but fail the purpose of the agent. Evaluate authorized completion and unauthorized effects against the same task distribution and tool access.
This is a benchmark with controlled attacks and varying models. It does not establish a universal ranking of every production agent defense or that every requested harmful call was executed. Audit both sides of each task: permitted end state and independently verified side effects, including rejected actions and tool availability.
Evidence reconciliation and effect boundary. The study measures both the attacker’s objective and the original user task, avoiding an apparent security victory in which every useful action is refused. Its scenarios differ from static benchmark suites because the attacker-controlled data and permission assumptions are built from more complex agent workflows. The ranking depends on the tested model, defense and attack budget; the measured decrease in attack success cannot be translated into universal prevention of unauthorized effects.
Mechanism and trust boundary
- 01
The study differentiates whether the attacker can reach an injection surface, whether an unauthorized action is proposed and whether it is actually performed. A reported zero attack rate is not meaningful without the fraction of useful tasks the defender still completes.
- 02
A guard that simply denies the workflow can have excellent attack scores but fail the purpose of the agent. Evaluate authorized completion and unauthorized effects against the same task distribution and tool access.
Claims & evidence
CLM-ENR243-019-DETAILMetaPI adapts AppWorld, VitaBench and Terminal-Bench utility tasks into indirect prompt-injection tests and reports that none of its evaluated defenses simultaneously maintains strong utility and security under its adaptive attack settings.supported
Basis: reported finding
- supportsMetaPI: Constructing Prompt Injection Benchmarks from LLM Agent Benchmarkprimary disclosure
MetaPI paper, Sections 4 and 5 Results; utility/attack-success comparison across defense configurations
CLM-R242-METAPI-DEFENSE-UTILITY-ATTRITION-LIMITSThis is a benchmark with controlled attacks and varying models. It does not establish a universal ranking of every production agent defense or that every requested harmful call was executed.supported
Basis: reported finding
- supportsMetaPI: Constructing Prompt Injection Benchmarks from LLM Agent Benchmarkprimary disclosure
Abstract and AppWorld, VitaBench, Terminal-Bench adaptive attack evaluations
CLM-R242-METAPI-DEFENSE-UTILITY-ATTRITION-MECHANISMMetaPI converts realistic agent utility tasks into adaptive prompt-injection evaluations, reusing AppWorld, VitaBench and Terminal-Bench environments. Its tests exposed cases where a defense appeared to block attack objectives while legitimate task completion also collapsed.supported
Basis: reported finding
- supportsMetaPI: Constructing Prompt Injection Benchmarks from LLM Agent Benchmarkprimary disclosure
Abstract and AppWorld, VitaBench, Terminal-Bench adaptive attack evaluations
Implications
A guard that simply denies the workflow can have excellent attack scores but fail the purpose of the agent. Evaluate authorized completion and unauthorized effects against the same task distribution and tool access.
Controls and mitigations
- Audit both sides of each task: permitted end state and independently verified side effects, including rejected actions and tool availability.
Unknowns and contradictions
- This is a benchmark with controlled attacks and varying models. It does not establish a universal ranking of every production agent defense or that every requested harmful call was executed.
Sources and citation
Material revision history
- Nov 13, 2026 · Canonical change recorded · new in release · revision 244