Multimodal-SEA: Benchmarking AI Agent Security under Multimodal Social Engineering Attacks

Evidence record

Read the original source ↗

Claim-level citations (3)

• supportsForged multimodal evidence made agents verify more and fall for more attacks: Adding multimodal evidence raised average attack success from 19.9% to 29.2% and measured verification coverage from 54.7% to 65.1% in the evaluated configurations.

  Section 5.1 Effect of Multimodal Content, multimodal ablation and verification tables

• supportsForged multimodal evidence made agents verify more and fall for more attacks: The controlled Multimodal-SEA evaluation reports 32.1% incorrect refusal of benign requests for Claude Opus 5 in its separate legitimate-task evaluation; low attacker success did not mean zero utility cost.

  Introduction and Experiments: benign tasks and over-refusal results

• supportsForged multimodal evidence made agents verify more and fall for more attacks: The benchmark contains 608 tasks across four difficulty levels, 14 domains and 152 social-engineering scenarios, using simulated tools and state checks.

  Abstract and Section 2 Benchmark Construction