Analysis · DiggingBeagle record
Five benchmark percentages that answer five different security questions
AuthorizationBench, Carryover, AutoRedClaw, BenchDoor and RegretBench measure unauthorized actions, planted instructions, persistent attack scenarios, model backdoors and restoration. None supplies a universal agent-compromise rate.
- Published
- Oct 23, 2026
Overview
A benchmark can report a high success rate without measuring the event a defender ultimately cares about. Some tests count stored malicious instructions, some count model outputs, some count unauthorized tool effects, and others measure whether changed files can be restored. Putting these figures on one ranking chart hides the point at which each experiment observed the system.
Write down the event and denominator before the percentage
| Study | Reported observation | What it is not |
|---|---|---|
| AuthorizationBench | 19.2% out-of-authorization behavior in 6,864 traces across 286 tasks and 24 evaluated settings | An estimated compromise rate of production deployments |
| Carryover | 101 attacks over twelve scenarios; persistence across five follow-up rounds conditional on successfully planted state | Unconditional five-session persistence of every injection |
| AutoRedClaw | 330 generated attack cases across seven harnesses and four models, with runtime-dependent attack and trigger rates | One comparable susceptibility score for all personal agents |
| BenchDoor | Positive benchmark gains in 30 selected training configurations with 82.33% to 100% held-out trigger success | Evidence that named deployed public models already contain those backdoors |
| RegretBench | 86.1% mean task completion, 15.2% backups before regret moments, 92.2% mean restoration similarity with BackUpper | Proof of exact recovery or authorization to make the original change |
The table is deliberately not a leaderboard. Its columns refer to different systems and security properties, and even the units vary: traces, scenarios, training configurations and restoration similarity.
The decisive point often occurs after the headline metric
A persistent injection has at least three observable stages: attacker-controlled material is accepted; the material is retained or reintroduced; and a later action violates the intended boundary. Carryover distinguishes planting from activation, while AutoRedClaw uses its own generated environments and harness-specific success definitions. Adding the two success percentages would neither estimate a joint risk nor yield a fair comparison.
BenchDoor operates earlier still, at training-data and evaluation incentives. A model could gain benchmark points in the authors' controlled settings while learning a trigger-controlled behavior. That result challenges the use of score improvement as an unqualified trust signal. It does not demonstrate a prompt injection into an operating agent or production victim compromise.
RegretBench moves the observation later, after a permitted coding task has changed state. Its reported 92.2% is character-level recovery similarity, not 92.2% of tasks returning every bit to the prior state. That distinction matters when evaluating whether rollback can actually undo consequential effects. Task success and successful recovery are separate axes even when both are measured on the same run.
A voting monitor adds yet another denominator
The correlated-approval study tests synthetic payment decisions. Its pre-registered worst-case condition failed for language-model monitors and succeeded with a deterministic rule engine on the sampled attacks. Its 0.9% versus 7.4% rejection comparison was measured at matched safety on legitimate examples. It is not an agent-attack-success figure and cannot be appended to the first five rows.
The transferable method is to record the actor's control, the defended boundary, the unit sampled, the success condition and whether the effect was actually observed. A dashboard that keeps these fields alongside each result can compare evidence quality without inventing a single "secure agent" number. The studies are useful precisely because they expose different failure stages rather than converging on one score.