An agent may produce the requested artifact while violating a required tool protocol, lifecycle rule or verification step. TraceTrial scores context, execution, governance, lifecycle, observability, tools and verification alongside task completion, making different failure classes visible. TraceTrial-Bench evaluated 3,420 runs across 95 executable tasks,…
Inspect the ClaimsCase · DiggingBeagle record
TraceTrial separates task completion from agent harness correctness
The study of 3,420 agent trajectories finds 31.1 percent of runs completing the task while failing at least one surrounding harness criterion. This is not a count of unlawful or unauthorized acts.
Scope
The composite 31.1 percent figure includes failures across several harness dimensions and must not be reworded as unauthorized access or exposed secrets. Task composition and evaluator design drive the observed rate.
Assessment profile
Assessment method- Setting
- evaluation
- Exploitation
- demonstrated
- Evidence
- Grade B
- Remediation
- not applicable
- AI role
- AGAINST AI
Basis and provenance
Enrich update: CLM-ENRICH-20261027-35 cites SRC-2026-CP21-TRACETRIAL-HARNESS-CORRECTNESS at Abstract, seven components and trajectory tracing evaluation. New evidence concerns the bounded reported mechanism, prerequisite or outcome; setting evaluation; exploit status demonstrated; remediation status not_applicable. No independently comparable impact scale, affected production population or universal exploit success measure was established, so the impact band remains null. Original Claims remain individually attributable and any observed incident or simulated result is distinguished in the narrative.
Assessed Oct 27, 2026 using diggingbeagle.assessment/1.
Evidence basis: CLM-ENRICH-20261027-35 · TraceTrial-Bench: executable correctness criteria for AI-agent harnesses
At a glance
This matters for security audits that confuse working output with permission-correct operation.
Read the implicationsThe composite 31.1 percent figure includes failures across several harness dimensions and must not be reworded as unauthorized access or exposed secrets. Task composition and evaluator design drive the observed rate.
Limits and uncertaintyFull account
An agent may produce the requested artifact while violating a required tool protocol, lifecycle rule or verification step. TraceTrial scores context, execution, governance, lifecycle, observability, tools and verification alongside task completion, making different failure classes visible.
TraceTrial-Bench evaluated 3,420 runs across 95 executable tasks, four harnesses and nine models. The authors report 31.1 percent task success combined with failure of at least one harness criterion. The composite 31.1 percent figure includes failures across several harness dimensions and must not be reworded as unauthorized access or exposed secrets. Task composition and evaluator design drive the observed rate.
Instrument effects and trace-level conditions independently of a final success bit, and require evidence for state transitions and authorization before crediting a successful run. This matters for security audits that confuse working output with permission-correct operation. Primary evidence.
Additional evidence and trust boundary
A passing artifact can coexist with an invisible policy violation, an unacknowledged external effect or a missing verification step. By grading trace-level behavior independently, the harness benchmark can distinguish model reasoning errors from orchestration and tool-enforcement defects. The reported 31.1 percent composite failures do not mean that 31.1 percent of evaluated runs committed an unauthorized real-world action. Supporting record.
Timeline
- Oct 23, 2026disclosure
Public disclosure
Claims & evidence
CLM-ENRICH-20261027-35TraceTrial-Bench defines seven separate harness dimensions, including context, governance, lifecycle and verification, and uses execution traces to attribute failure rather than equating a completed task with an authorized workflow.supported
Basis: reported finding
- supportsTraceTrial-Bench: executable correctness criteria for AI-agent harnessesprimary disclosure
Abstract, seven components and trajectory tracing evaluation
CLM-CP21-TRACETRIAL-HARNESS-CORRECTNESS-1TraceTrial-Bench evaluated 3,420 runs across 95 executable tasks, four harnesses and nine models. The authors report 31.1 percent task success combined with failure of at least one harness criterion.supported
Basis: reported finding
- supportsTraceTrial-Bench: executable correctness criteria for AI-agent harnessesprimary disclosure
Abstract; harness criteria and 3,420-trajectory evaluation
Implications
This matters for security audits that confuse working output with permission-correct operation.
Controls and mitigations
- Instrument effects and trace-level conditions independently of a final success bit, and require evidence for state transitions and authorization before crediting a successful run.
Unknowns and contradictions
- The composite 31.1 percent figure includes failures across several harness dimensions and must not be reworded as unauthorized access or exposed secrets. Task composition and evaluator design drive the observed rate.
Sources and citation
Material revision history
- Oct 27, 2026 · Canonical change recorded · new in release · revision 196