Multimodal-SEA tests agents acting on a user's behalf against social engineering supported by forged documents, images and websites. The benchmark contains 608 tasks derived from 152 scenarios across 14 domains and four difficulty levels. Its tools and outcome checks operate in simulated environments, not real customer accounts. Adding multimodal artifacts…
Inspect the ClaimsCase · DiggingBeagle record
Forged multimodal evidence made agents verify more and fall for more attacks
Multimodal-SEA reported attack success rising from 19.9% to 29.2% when forged multimodal artifacts were added. Verification coverage also rose, showing that performing a check and obtaining independent evidence are not equivalent.
Public disclosure: Nov 5, 2026
Scope
Research reported in a controlled research or disclosure setting. Exact prerequisites, versions and absence of documented exploitation must be preserved.
Assessment profile
Assessment method- Setting
- evaluation
- Exploitation
- demonstrated
- Evidence
- Grade C
- Remediation
- not applicable
- AI role
- AGAINST AI
Basis and provenance
The cited first-party advisory or research establishes the stated mechanism only within its documented conditions. No unrelated production compromise, extended victim scope, financial harm or universal mitigation is inferred.
Assessed Nov 9, 2026 using diggingbeagle.assessment/1.
Evidence basis: CLM-MSEA-SIMULATED-TASK-SIZE · CLM-MSEA-MULTIMODAL-RESULT · Multimodal-SEA: Benchmarking AI Agent Security under Multimodal Social Engineering Attacks
At a glance
The study separates how often an agent tries to verify from whether its verification is independent of attacker-controlled evidence.
Read the implicationsNo real victim payments, true prevalence of fraud or causal claim across production agent products; model sets and cultural scenarios are nonidentical.
Limits and uncertaintyFull account
Multimodal-SEA tests agents acting on a user's behalf against social engineering supported by forged documents, images and websites. The benchmark contains 608 tasks derived from 152 scenarios across 14 domains and four difficulty levels. Its tools and outcome checks operate in simulated environments, not real customer accounts.
Adding multimodal artifacts raised average attack success from 19.9% to 29.2% in the evaluated configurations, while measured verification coverage increased from 54.7% to 65.1%. More checks did not necessarily provide independent corroboration: the content presented as evidence was itself part of the deception.
Refusal can conceal another cost
In the separate legitimate-task evaluation, Claude Opus 5 incorrectly refused 32.1% of benign requests. That rate has a different population from the multimodal attack ablation and should not be combined with it. A low observed attacker success rate can coexist with poor completion of authorized work.
The study motivates checking the provenance and independence of verification, alongside actual tool effects and legitimate-task utility. It does not prove that every model became less safe under multimodal input, that stronger safety instructions are useless, or that the simulated losses occurred in production. These are conditional benchmark results, not an observed fraud campaign.
Mechanism and trust boundary
- 01
A personal agent receives forged evidence in a low-trust screenshot, document or website related to a legitimate task.
- 02
The forged artifact corroborates a deceptive request or impersonated claim and may influence an apparent verification action.
- 03
The agent executes a payment, disclosure or other tool effect contrary to the original trusted user intent in the simulated environment.
Claims & evidence
CLM-MSEA-MULTIMODAL-RESULTAdding multimodal evidence raised average attack success from 19.9% to 29.2% and measured verification coverage from 54.7% to 65.1% in the evaluated configurations.supported
Basis: reported finding
- supportsMultimodal-SEA: Benchmarking AI Agent Security under Multimodal Social Engineering Attacksprimary disclosure
Section 5.1 Effect of Multimodal Content, multimodal ablation and verification tables
CLM-ENR231-MSEA-OVERREFUSALThe controlled Multimodal-SEA evaluation reports 32.1% incorrect refusal of benign requests for Claude Opus 5 in its separate legitimate-task evaluation; low attacker success did not mean zero utility cost.supported
Basis: reported finding
- supportsMultimodal-SEA: Benchmarking AI Agent Security under Multimodal Social Engineering Attacksprimary disclosure
Introduction and Experiments: benign tasks and over-refusal results
CLM-MSEA-SIMULATED-TASK-SIZEThe benchmark contains 608 tasks across four difficulty levels, 14 domains and 152 social-engineering scenarios, using simulated tools and state checks.supported
Basis: reported finding
- supportsMultimodal-SEA: Benchmarking AI Agent Security under Multimodal Social Engineering Attacksprimary disclosure
Abstract and Section 2 Benchmark Construction
Implications
The study separates how often an agent tries to verify from whether its verification is independent of attacker-controlled evidence.
Controls and mitigations
- Require independent origin checks for verification sources and compare final tool effects with the authenticated request, rather than counting verification steps alone.
Unknowns and contradictions
- No real victim payments, true prevalence of fraud or causal claim across production agent products; model sets and cultural scenarios are nonidentical.
Sources and citation
Material revision history
- Nov 9, 2026 · Published version · first publication · revision 233