Evidence · DiggingBeagle record
MetaPI: Constructing Prompt Injection Benchmarks from LLM Agent Benchmark
Primary or origin disclosure used to qualify the underlying mechanism, stated prerequisites and limits. arXiv2611.10026: 150 sampled tasks AppWorld/VitaBench/Terminal; 183 attacks on107 tasks, 104 deflection/79 injection, 94 necessary read/89 incidental. Baseline ASR 50%,45.9%,44.4%. CaMeL AppWorld attack0% but utility33.7%; Terminal0% attack and0% utility => no reachable attacks. Some meta-align comparisons other backbone. Effective ASR differs net.
- Source role
- primary disclosure
Evidence record
Primary or origin disclosure used to qualify the underlying mechanism, stated prerequisites and limits. arXiv2611.10026: 150 sampled tasks AppWorld/VitaBench/Terminal; 183 attacks on107 tasks, 104 deflection/79 injection, 94 necessary read/89 incidental. Baseline ASR 50%,45.9%,44.4%. CaMeL AppWorld attack0% but utility33.7%; Terminal0% attack and0% utility => no reachable attacks. Some meta-align comparisons other backbone. Effective ASR differs net.
Claim-level citations (3)
- supportsMetaPI shows low injection success can coincide with a defense that cannot finish tasks: MetaPI adapts AppWorld, VitaBench and Terminal-Bench utility tasks into indirect prompt-injection tests and reports that none of its evaluated defenses simultaneously maintains strong utility and security under its adaptive attack settings.
MetaPI paper, Sections 4 and 5 Results; utility/attack-success comparison across defense configurations
- supportsMetaPI shows low injection success can coincide with a defense that cannot finish tasks: This is a benchmark with controlled attacks and varying models. It does not establish a universal ranking of every production agent defense or that every requested harmful call was executed.
Abstract and AppWorld, VitaBench, Terminal-Bench adaptive attack evaluations
- supportsMetaPI shows low injection success can coincide with a defense that cannot finish tasks: MetaPI converts realistic agent utility tasks into adaptive prompt-injection evaluations, reusing AppWorld, VitaBench and Terminal-Bench environments. Its tests exposed cases where a defense appeared to block attack objectives while legitimate task completion also collapsed.
Abstract and AppWorld, VitaBench, Terminal-Bench adaptive attack evaluations