Shortcut Sensitivity: lexical shortcuts in agent security evaluation

Evidence record

Study measures tool-name/command-length shortcut prediction and degradation on unseen domains or interaction-weighted scoring, rather than demonstrated live compromise.

Read the original source ↗

Claim-level citations (4)

• supportsAgent security monitor benchmarks rewarded command-length and tool-name shortcuts: In tested LinuxArena data, lexical or command-length features achieved high classification AUROC without causal analysis of a command’s harmful effect.

  Lexical classifier and command-length controls

• supportsAgent security monitor benchmarks rewarded command-length and tool-name shortcuts: A tool-name shortcut classifier scored around 0.901 F1 on random-fold AgentDojo evaluation but 0.495 under unseen-domain split, highlighting overfit.

  AgentDojo domain-shift experiments and label shortcut tables

• supportsAgent security monitor benchmarks rewarded command-length and tool-name shortcuts: The study reports substantial changes in aggregate F1 depending on whether AgentHarm is weighted by individual row or full interaction.

  AgentHarm interaction weighting comparison

• supportsAgent security monitor benchmarks rewarded command-length and tool-name shortcuts: An exact-tool-name lookup reached F1 0.901 on random folds but 0.495 on held-out tool domains, demonstrating dataset dependence rather than universal guard accuracy.

  Shortcut sensitivity paper, contributions and domain holdout comparison