Screwtape benchmark exposed validity gaps in coding-agent monitor evasions

Redwood Research took 94 traces from real coding-agent work and used a separate attacker-generation process to insert plausible-seeming malicious actions into the histories. The hypothetical misbehavior included credential exposure, unauthorized deployments and sabotage. The monitors examined edited transcripts, not verified real-world malicious executions.

Across 252 candidate generations, 35 examples passed a local validity test while evading a proxy monitor. Seventeen also evaded a held-out developer monitor, but human reviewers accepted only six as valid attacks. This distinction is central: classifier evasion is not the same as a coherent, executable unauthorized action.

Security evaluations need two independent witnesses: a detector that observes the proposed activity and an adversarial validity test that establishes the activity actually happened under the stated tool semantics. Monitor-success numbers alone overcount attack outcomes when fabricated histories contain incompatible steps or impossible tool effects.

The red-team funnel changes its denominator at least twice: 94 base coding-agent traces produced 252 generations; 35 edited candidate attacks met local conditions, 17 evaded a held-out monitor, and only six survived human attack-validity review. A verifier could be fooled by an impossible action inserted into a plausible-looking transcript, so monitor performance must not be reported as an exploited deployment success rate. Transfer also varied substantially across different proxy monitors. The study's finding is evidence about action-history evaluation design and false confidence under partial observation, not proof that a production credential was taken.

These traces expose a second evidence boundary: the monitor saw plausible edited histories, not completed intrusions. The six human-accepted edits therefore support a conditional monitor-red-team finding; they do not establish that six target systems were attacked. The associated execution-evidence pattern separates a monitor's classification from the validity of the supplied transcript and from independently witnessed host behavior.