Why 94% tamper acceptance, six monitor evasions and missed processes cannot share a score

Three October research projects evaluated AI agent security using three incompatible observations: the final response, an edited record of agent actions, and underlying operating-system events. Their headline numbers should not be pooled as an 'agent safety rate'. They answer different questions about what ran, what an observer might believe, and what the machine actually did.

In the controlled DIB-Bench study, researchers changed one part of a model deployment while maintaining plausible outputs. Their six tamper surfaces included the model, prompt, tools, retrieval, dependencies and output postprocessor. Across 7,560 scored examples, standard rule scoring accepted 88.6% of the tampered output population, Qwen3-8B judging accepted 89.3%, and GPT-5-nano accepted 94.1%. A separate identity channel rejected all the altered configurations declared in that experimental setup.

The apparent paradox is procedural. A judge asked whether an answer is satisfactory may legitimately accept correct-looking text even though an unapproved retrieval index or tool implementation produced it. Evaluating answer quality cannot establish deployment identity. Conversely, hashing a manifest establishes the identity of listed artifacts, not whether they are safe. The paper explicitly demonstrates that an undeclared component can defeat an incomplete identity manifest. These are constructed benchmark populations, not percentages of real-world compromised servers. The DIB-Bench Case separates the two checks.

Redwood Research's Screwtape study edited 94 real coding-agent transcripts across 252 generation runs to simulate prohibited actions and optimize against a monitor. It produced 35 candidate edits passing an automated validity check and evading a proxy monitor. Seventeen also escaped a held-out developer monitor. Human reviewers accepted just six of those seventeen as consistent and severe enough for the stated threat contract.

The denominator changes during this funnel: 94 source traces are not 252 independent exploits, and 17 apparent evasions are not 17 proven attacks. The malicious actions were not executed in a live system; the edited transcripts were stress tests of an action-only monitor. The authors' separate test deliberately introduced 63 inconsistent edits. Their verifier rejected 55, correctly allowing all 44 unchanged controls, but accepted eight known inconsistencies. Automated transcript plausibility is itself a measurement with false acceptance. The Screwtape Case records these limits.

The ExecWeave paper built a graph connecting semantic agent tool calls with observed process, file and network events. It explicitly distinguishes a recorded action from an inferred correlation. In one 30-run test of short-lived child processes, a portable 20-millisecond polling interval observed 18 children, while 500-millisecond polling observed none. These results concern a particular host, requested sleep and event collector, not a universal operating-system detection probability.

The cost was also concrete. For three paired Linux workloads, the paper reported median wall-time overheads of 408.8%, 85.2% and 62.3%. More intensive observation may catch more effects, but it is not free and may still miss the very short processes used in an attack. The researchers showed a further difficulty: if the true process was never collected, a correlator can confidently associate a tool call with the wrong observed process. Abstaining when visible evidence is ambiguous does not repair invisible evidence. The ExecWeave Case keeps this qualification explicit.

The order matters when interpreting a security review. The declared model and tool versions identify what was supposed to run. A trace then records what the agent and tools said occurred. Host and service-side logs can corroborate side effects, subject to the collector's completeness. A model-generated final report is useful context but cannot replace any of those measurements. Nor can more forceful language from an evaluator close a gap created by missing deployment identities or unobserved processes.

These studies do not show that all monitors fail or that runtime attestation guarantees safety. They show where particular observations stop being evidence for a stronger claim. A defensible agent evaluation should publish the unit tested, its observation channel, the intervention or ground truth, and the limits of detection before it advertises a success percentage.

An evaluator can hold the user task constant while changing the component that supplies the evidence. If the adversary substitutes the deployed model or a retriever, compare signed or reproducible component identity with the declared manifest. If it edits a transcript, first validate that the trace is internally consistent before asking whether a monitor would flag it. If it controls short-lived subprocesses, measure the collector's miss rate instead of inferring their absence from missing events. Each exercise needs different attacker permissions and a different independent ground truth. These distinctions define the agent execution evidence triangulation pattern.

A combined score would erase those permissions and observation gaps. A more useful test report separates the deployment identity denominator, the number of edited traces that survive validity review, and the effects directly witnessed by service or host telemetry. Even three positive results do not establish that the underlying task authorization was correct; the policy-attested execution boundary remains another independent control.