AgentCIScope tests whether a multi-tool assistant discloses protected contextual information to unauthorized recipients while completing otherwise legitimate work. Its contracts identify protected attributes, source and destination participants, and task-allowed data flows. The reported leakage varies substantially with agent, model, framework, environment…
Inspect the ClaimsCase · DiggingBeagle record
AgentCIScope tests privacy boundary failures in cross-application workflows
Across 936 generated executable agent tasks, the authors observed 32.5% to 94.4% single-run task-level disclosure rates under their benchmark contracts.
Scope
Controlled benchmark combining public-derived and synthetic privacy cases across 11 simulated application environments and 12 evaluated agents; not evidence of a breach in real user accounts.
Assessment profile
Assessment method- Setting
- evaluation
- Exploitation
- demonstrated
- Evidence
- Grade B
- AI role
- AGAINST AI
Basis and provenance
Original paper directly inspected, including methodology and limited human checks. Disclosures were measured in synthetic or public-derived controlled task environments, not real account breaches. Exploit status reflects observed benchmark disclosures only; no real-world impact is quantified. No deployed remediation is established.
Assessed Oct 16, 2026 using diggingbeagle.assessment/1.
Evidence basis: CLM-ACIS-936 · CLM-ACIS-LEAKS · CLM-ACIS-ATTRIBUTES · CLM-ACIS-JUDGE-HUMAN · AgentCIScope: Grounded Contextual Integrity Evaluation for LLM Agents
At a glance
Data-flow policy must be enforced against the recipient and the purpose of transfer, not merely against a global definition of sensitive strings.
Read the implicationsBenchmark policies, personas and some cases are synthesized or validated with model assistance; measured rates depend on its constructed environments and disclosure rubric. The paper does not establish actual leakage rates for production instances of the named agents or prove an adversarial compromise. The authors focus on single-agent workflows;…
Limits and uncertaintyFull account
AgentCIScope tests whether a multi-tool assistant discloses protected contextual information to unauthorized recipients while completing otherwise legitimate work. Its contracts identify protected attributes, source and destination participants, and task-allowed data flows. The reported leakage varies substantially with agent, model, framework, environment and attribute type; higher task completion does not entail compliant information flow. The benchmark relies partly on LLM-generated content and model-assisted validation, and should not be extrapolated as a deployment-wide privacy incident rate. The original paper measures a specific rule: whether an attribute moves from a source application to an allowed destination for the stated task. The authors separately count an agent completing the requested work and an agent respecting those disclosure restrictions. Calendar and messaging destinations matter because a harmless field in its original context can become an inappropriate disclosure in another. Human cross-checks cover a small subset, not all 936 evaluated tasks. The study does not test a finished access-control product or prove that production agents leak at the same rate.
Mechanism and trust boundary
- 01
Agent receives ordinary workspace task
The user asks the agent to execute a valid task across mail, documents or similar applications.
Boundary: user request to multi-app tools
- 02
Agent retrieves protected context
Its tools make task-associated private fields available along with necessary public context.
Boundary: source app to agent context
- 03
Agent crosses disclosure boundary
The agent includes protected details in output or messages to participants not authorized by the benchmark contract.
Boundary: agent context to recipient app
Timeline
- Oct 13, 2026disclosure
Public disclosure
Claims & evidence
CLM-ACIS-936The authors constructed 936 executable tasks with 2,771 protected attributes over 11 application environments and evaluated 12 agents.supported
Basis: reported finding
- supportsAgentCIScope: Grounded Contextual Integrity Evaluation for LLM Agentsprimary disclosure
Introduction, contributions; section 3, benchmark construction
CLM-ACIS-LEAKSThe paper reports single-run task-level leakage rates ranging from 32.5% to 94.4%, and an additional 4 to 19 percentage points when each task is repeated five times.supported
Basis: reported finding
- supportsAgentCIScope: Grounded Contextual Integrity Evaluation for LLM Agentsprimary disclosure
Abstract and Introduction, agent-level leakage rates
CLM-ACIS-ATTRIBUTESThe authors report higher average leakage for personal preferences (84.7%) and booking details (76.5%) than for credit-card data (17.1%).supported
Basis: reported finding
- supportsAgentCIScope: Grounded Contextual Integrity Evaluation for LLM Agentsprimary disclosure
Abstract and Introduction, attribute categories
CLM-ACIS-JUDGE-HUMANIn a 50-task human-validation sample, the model-based leakage judge agreed with human annotations on 92.0% of task-level labels and 95.5% of attribute-level labels. This is validation of the measurement method on a subset, not an audit of all 936 tasks.supported
Basis: reported finding
- supportsAgentCIScope: Grounded Contextual Integrity Evaluation for LLM Agentsprimary disclosure
Appendix D.2, Table 5, 50-task and 157-attribute human validation
Implications
Data-flow policy must be enforced against the recipient and the purpose of transfer, not merely against a global definition of sensitive strings.
Controls and mitigations
- Define recipient-specific disclosure rules before an agent transfers workspace data. Enforce them at tool-write or message-send boundaries so task completion cannot override them. This is an engineering implication of the measured failure mode, not a defense evaluated by the paper.
- Log destination, recipient, field, and task purpose for outbound agent transfers so a reviewer can check the contextual rule. Avoid treating keyword-based secret filters as equivalent to recipient authorization.
Unknowns and contradictions
- Benchmark policies, personas and some cases are synthesized or validated with model assistance; measured rates depend on its constructed environments and disclosure rubric.
- The paper does not establish actual leakage rates for production instances of the named agents or prove an adversarial compromise.
- The authors focus on single-agent workflows; multi-agent delegation, real user consent, and end-to-end production enforcement were not evaluated.
- Human validation was performed on a 50-task sample using one agent setup. It cannot independently validate every benchmark policy or trajectory.
Sources and citation
Material revision history
- Oct 16, 2026 · Canonical change recorded · new in release · revision 157