A durable hash can identify exactly which log or document was preserved. It cannot, by itself, establish that the record concerned the correct principal, existed when an agent acted or demonstrated the claimed outcome. AgentEval used counterfactual changes to evidence views to probe situations where the bytes were genuine but the narrative bound them to a…
Inspect the ClaimsCase · DiggingBeagle record
Matching evidence hashes failed to prove evidence supported the evaluated action
AgentEval showed that cryptographically identifiable evidence could still be attached to the wrong agent decision or action.
Scope
Small controlled AgentEval study with 48 counterfactual evidence views and reported binding errors in 36 of them. No independently verified production forensic deployment or incident population.
Assessment profile
Assessment method- Setting
- evaluation
- Exploitation
- demonstrated
- Impact
- Band 1 of 4
- Evidence
- Grade B
- Remediation
- not applicable
- AI role
- AGAINST AI
Basis and provenance
The AgentEval paper uses 48 constructed counterfactual evidence views and reports 36 mistaken bindings. A correct digest proves byte identity, not evidentiary relevance or authority. Its bounded experimental setup and lack of independent operational replication support a modest score.
Impact basis: A synthetic audit can misattribute correctly hashed evidence to the wrong action or decision. This undermines evaluation integrity but the study does not demonstrate a failed real-world forensic or financial decision.
Assessed Oct 19, 2026 using diggingbeagle.assessment/1.
Evidence basis: CLM-AGENTEVAL-EVIDENCE-MISBIND · AgentEval: evidence-bounded assessment of stateful agents
At a glance
The research also distinguishes interrupted episodes and successful subgoals from a completed authorized task. A reviewer that observes only the final output can miss meaningful partial work or unauthorized side effects.
Read the implicationsThe study used a small set of constructed tool environments and does not certify a production forensic system. The reported 36 binding errors among 48 counterfactual views are a benchmark result, not a prevalence estimate.
Limits and uncertaintyFull account
A durable hash can identify exactly which log or document was preserved. It cannot, by itself, establish that the record concerned the correct principal, existed when an agent acted or demonstrated the claimed outcome. AgentEval used counterfactual changes to evidence views to probe situations where the bytes were genuine but the narrative bound them to a different action or decision.
This exposes a problem in reports that summarize an entire trajectory under one success label. An agent may have completed a subgoal while leaving another step undone, or a report may cite an observation from the wrong phase. The study's 36 errors among 48 counterfactual views are a result within deliberately constructed testing conditions, not a measured fraction of production AI audit trails.
A stronger evidence record links source content to actor identity, action arguments, observed external effects and event time. When the action and observed effect differ, the discrepancy should remain visible rather than being reconciled into a polished final-answer narrative. Document hashes are necessary for tamper evidence, but they do not solve attribution or causal misbinding.
Claims & evidence
CLM-AGENTEVAL-EVIDENCE-MISBINDAgentEval tested stateful agent reports against controlled evidence views and found cases where the referenced bytes existed but were bound to the wrong decision or action.supported
Basis: reported finding
- supportsAgentEval: evidence-bounded assessment of stateful agentsprimary disclosure
Abstract; counterfactual evidence views and result scoring
Implications
The research also distinguishes interrupted episodes and successful subgoals from a completed authorized task. A reviewer that observes only the final output can miss meaningful partial work or unauthorized side effects.
Controls and mitigations
- Bind evidence to principal, versioned state, acquisition time and specific external effect; verify actual outcomes separately from document hashes.
Unknowns and contradictions
- The study used a small set of constructed tool environments and does not certify a production forensic system. The reported 36 binding errors among 48 counterfactual views are a benchmark result, not a prevalence estimate.
Sources and citation
Material revision history
- Oct 19, 2026 · Published version · first publication · revision 174
- Oct 19, 2026 · Canonical change recorded · material revision · revision 175Public fields changed: assessment_profile