Topic · DiggingBeagle record
Agent execution evidence triangulation
Model outputs, monitor transcripts, deployment manifests and observed host events answer different questions; none alone proves what an agent was authorized to run or what effects completed.
- Topic kind
- pattern
Definition & limits
In AI-agent investigations, four superficially similar artifacts have different evidentiary force. An output evaluation tests whether a response appears good; a deployment manifest identifies declared model, prompts, tools, retrieval components and dependencies; an agent action trace records tool calls or purported calls; and host or remote-service telemetry may observe effects that the trace misrepresents. A defensible assertion must specify which channel is measured and whether its collection is complete.
DIB-Bench deliberately tampered with model-serving components and showed that output-only judges frequently accepted plausible results; the benchmark measures acceptance of engineered substitutions, not compromised production prevalence. Screwtape tested edited transcripts for monitor evasion, but only six of seventeen held-out apparent evasions were human-accepted and those edits did not execute harmful effects. ExecWeave kept semantic traces separate from process events, while exposing a missing-process problem under slower polling. These are complementary limitations, not comparable success-rate denominators.
A checked component manifest can establish the identities of components it actually covers but not their safety. A believable trace can establish what a monitor saw but not that an action occurred. An OS observation can corroborate effects but cannot prove events never happened when its sensors miss short-lived processes. Investigations should link authorized intent, declared build, action log, and independently witnessed side effect without silently promoting inference into observation. This differs from general agent authority, which asks what may be done, and from report-state recoverability, which asks what state can be reconstructed after interruption.
Examples
- A deployed retrieval component is swapped while the judge still accepts the final answer in DIB-Bench.
- An edited coding-agent trace evades a monitor but fails human validity review in Screwtape.
- A short-lived subprocess is missing from ExecWeave portable polling, so an apparently unique correlation may be wrong.