News · DiggingBeagle record

Cross-app agent benchmark separates completed work from permitted disclosure

AgentCIScope finds task-level policy leakage in a controlled 936-task evaluation, while warning against treating benchmark outcomes as live customer incidents.

Published
Oct 16, 2026

The report

An assistant can complete a calendar or messaging task and still send the wrong information to the wrong recipient. AgentCIScope, posted on October 13, tests that separation using 936 executable tasks, 2,771 protected attributes and 11 application environments. The authors evaluated 12 agent setups and report single-run task-level leakage rates from 32.5% to 94.4% under their scenario-specific disclosure contracts. Those percentages are measurements from a constructed benchmark, not a count of affected people or production breaches.

The rule being tested is more precise than whether an output contains a password. The same personal detail may be appropriate in one exchange and prohibited in another, depending on purpose and recipient. The AgentCIScope study therefore separates task completion from whether protected fields crossed the stated boundary. It also reports differences by attribute type and destination application, which a single overall rate would obscure.

The authors combined public-derived and synthetic scenarios, filtered for difficult test cases and used model-assisted judging, with a smaller human validation sample. That construction makes the evaluation useful for comparing failure modes but limits external prevalence claims. A recipient-aware write or send check is one possible architectural response, not a mitigation experimentally proved by this paper.

Source: AgentCIScope primary paper.

Research behind this