Agent security monitor benchmarks rewarded command-length and tool-name shortcuts

An agent security guard can appear to understand harmful tool behavior while relying on superficial features that correlate with benchmark labels. The Shortcut Sensitivity study tests this possibility by constructing simple predictors from command length and tool names, then changing data splits, names and row weighting without changing the substantive authorization question.

On selected LinuxArena data, command length alone achieved an AUROC around 0.91, and lexical features around 0.934, despite not understanding the requested side effect. A tool-name lookup classifier reached 0.901 F1 on random-fold AgentDojo evaluation but only 0.495 on held-out tool domains. In another example, one AgentHarm guard's F1 changed from 0.902 when rows were weighted individually to 0.719 when entire interactions were weighted. These figures are specific to different experiments and cannot be merged into a general guard failure rate.

The evidence demonstrates shortcut sensitivity and a construct-validity problem, not universal production exploitability. Some tested guards were comparatively resistant to particular changes, and a classifier output is still not proof that a forbidden external action occurred. Evaluation should use unseen tools, interaction-level truth, controlled harmless metadata edits and an independent check of the actual authorized effect.