DEPUTYBENCH: Evaluating Contextual Safety in Enterprise Agents

Evidence record

Primary or origin disclosure used to qualify the underlying mechanism, stated prerequisites and limits. arXiv2611.13355: 79 base enterprise tasks/375 instances/12 models, legitimate twins differ permission state. 28 derived tasks, 5,058 runs across six models: explicit denial 17/486 restricted actions 3.5%, missing evidence100/972 10.3%, +6.8pp CI1.9-13.0. Four privacy tasks account 61 of100; exclude them difference+1.8 and CI crosses0. Grader audit improved 178/284->266/284.

Read the original source ↗

Claim-level citations (3)

• supportsDeputyBench found missing authorization evidence riskier than an explicit refusal: DeputyBench contains 79 base items and 375 executable permission-context instances; the revised 12-model audit reported 88.0% pooled safety, 90.4% completion and 79.1% paired success, with scorer corrections affecting interpretation.

  DeputyBench, Abstract, Introduction and scorer-v2.1 results

• supportsDeputyBench found missing authorization evidence riskier than an explicit refusal: These are executable benchmark tasks, not observed corporate privacy violations, and grader corrections materially changed the measured outcomes.

  Abstract, 79 base items and masked-versus-denied permission experiment with confidence interval

• supportsDeputyBench found missing authorization evidence riskier than an explicit refusal: DEPUTYBENCH evaluates agents that have to combine ordinary enterprise tasks with underlying authorization records. In paired controlled situations, models made more restricted-action mistakes when the relevant permission evidence was absent than when it explicitly said no.

  Abstract, 79 base items and masked-versus-denied permission experiment with confidence interval