A model log saying that a tool call was requested cannot establish that a file was modified or an API transaction completed. AgentTriage reconciles runtime traces and effect evidence from heterogeneous environments under an explicit diagnosis contract. primary evidence turns that distinction into a measurable auditing method. The reported diagnosis and…
Inspect the ClaimsCase · DiggingBeagle record
AgentTriage required runtime evidence before calling an agent action completed
AgentTriage compares agent events across runtimes to distinguish intended tool calls from completed side effects and abstains when the evidentiary chain is absent.
Assessment profile
Assessment method- Setting
- evaluation
- Exploitation
- not established
- Evidence
- Grade C
- Remediation
- not applicable
- AI role
- WITH AI
Basis and provenance
Primary method paper reports evaluated forensic accuracy, with no independent live production qualification. The impact band remains unknown: available evidence does not establish actual deployed victim outcomes or a comparable measured loss.
Assessed Oct 26, 2026 using diggingbeagle.assessment/1.
Evidence basis: CLM-CP20-AGENTTRIAGE-EFFECTS · CLM-CP20-ENRICH-AGENTTRIAGE-EFFECTS · AgentTriage evidence-grounded tool-effect tracing across five runtimes
At a glance
The reported diagnosis and evidence-coverage figures are benchmark results; they do not guarantee that unavailable or altered production logs can always reconstruct the action.
Limits and uncertaintyFull account
A model log saying that a tool call was requested cannot establish that a file was modified or an API transaction completed. AgentTriage reconciles runtime traces and effect evidence from heterogeneous environments under an explicit diagnosis contract. primary evidence turns that distinction into a measurable auditing method.
The reported diagnosis and evidence-coverage figures are benchmark results; they do not guarantee that unavailable or altered production logs can always reconstruct the action.
Bind tool start, completion, retries, external acknowledgement and final state to stable IDs. If one required part is missing, mark the outcome unknown instead of converting the request into proof of effect.
The evidence chain has three separate identities: the invocation, its returned result, and any durable external effect. AgentTriage normalized 159 applicable executions across five runtimes and judged 120 cases. Its reported grounded joint diagnosis was 0.858, required-evidence coverage 0.966 and unsupported-claim rate 0.036. These are scored diagnoses against an adjudicated set, not evidence that every production tool effect can be recovered from missing or tampered telemetry. An auditor should mark the side effect indeterminate when it lacks an authoritative result or external receipt, even if the assistant has already claimed success.
Claims & evidence
CLM-CP20-AGENTTRIAGE-EFFECTSThe AgentTriage study describes a replayable diagnosis contract across five agent runtimes with groundable claims and mandatory abstention when evidence is incomplete.supported
Basis: reported finding
- supportsAgentTriage evidence-grounded tool-effect tracing across five runtimesrole not specified
Abstract; five-runtime replay; diagnosis evidence constraints
CLM-CP20-ENRICH-AGENTTRIAGE-EFFECTSAgentTriage reports 159 applicable executions across five runtimes; on 120 adjudicated cases it measured grounded joint diagnosis of 0.858, required-evidence coverage of 0.966 and unsupported-claim rate of 0.036.supported
Basis: reported finding
- supportsAgentTriage evidence-grounded tool-effect tracing across five runtimesrole not specified
Abstract; five-runtime normalized executions and 120 adjudicated cases; accuracy and coverage metrics
Unknowns and contradictions
- The reported diagnosis and evidence-coverage figures are benchmark results; they do not guarantee that unavailable or altered production logs can always reconstruct the action.
Sources and citation
Material revision history
- Oct 26, 2026 · Canonical change recorded · new in release · revision 193