AgentCIScope: Grounded Contextual Integrity Evaluation for LLM Agents

Evidence record

Original study of contextual-integrity constraints and cross-application information flows in multi-step agent tasks, with author-reported benchmark results.

Read the original source ↗

Claim-level citations (4)

• supportsAgentCIScope tests privacy boundary failures in cross-application workflows: The authors constructed 936 executable tasks with 2,771 protected attributes over 11 application environments and evaluated 12 agents.

  Introduction, contributions; section 3, benchmark construction

• supportsAgentCIScope tests privacy boundary failures in cross-application workflows: The paper reports single-run task-level leakage rates ranging from 32.5% to 94.4%, and an additional 4 to 19 percentage points when each task is repeated five times.

  Abstract and Introduction, agent-level leakage rates

• supportsAgentCIScope tests privacy boundary failures in cross-application workflows: The authors report higher average leakage for personal preferences (84.7%) and booking details (76.5%) than for credit-card data (17.1%).

  Abstract and Introduction, attribute categories

• supportsAgentCIScope tests privacy boundary failures in cross-application workflows: In a 50-task human-validation sample, the model-based leakage judge agreed with human annotations on 92.0% of task-level labels and 95.5% of attribute-level labels. This is validation of the measurement method on a subset, not an audit of all 936 tasks.

  Appendix D.2, Table 5, 50-task and 157-attribute human validation