ScamWorld interactive web-agent fraud benchmark, November 4 revision

Evidence record

Original arXiv v2 study: 363 reconstructed scam sites, 53 legitimate examples and six agent-model backbones.

Read the original source ↗

Claim-level citations (4)

• supportsWeb agents completed simulated scam payments even after identifying fraud warning signs: On 363 reconstructed scam websites, the six tested agent backbones took the fraud-completing final action in 11% to 35% of simulated task runs; the denominator is benchmark episodes, not real consumer theft.

  Abstract; Section 4.2 Main Results Table 2, All families

• supportsWeb agents completed simulated scam payments even after identifying fraud warning signs: The paper reports 28% deception in scenarios involving payment, versus 2% where the final action requested credentials or wallet data; those are scenario-conditioned benchmark outcomes.

  Section 4.2 Finding 2: payment versus credential requests

• supportsWeb agents completed simulated scam payments even after identifying fraud warning signs: Among 492 deceived benchmark episodes, 220 (45%) included a model warning about a website-specific risk before it nevertheless took the fraud-completing final action.

  Section 4.2 Finding 3, 220 of 492 deceived episodes

• supportsWeb agents completed simulated scam payments even after identifying fraud warning signs: In ScamWorld, agents explicitly asked to classify sites as scams identified about 97% of fraudulent websites, but task-executing agents stopped and reported suspicion on only about 75% of tasks, illustrating that diagnosis and action control are different evaluations.

  Section 4.3 Scam Identification, paragraph comparing 97% explicit identification with 75% execution warnings