Evidence · DiggingBeagle record
When Evaluators of AI Agents Disagree
Study of 2,160 tau2-bench episodes comparing native scorers, LLM judges, human review and commit-boundary checks.
- Published
- Nov 10, 2026
- Source role
- primary disclosure
Evidence record
Study of 2,160 tau2-bench episodes comparing native scorers, LLM judges, human review and commit-boundary checks.
Claim-level citations (2)
- supportsAn agent evaluator sometimes gave a passing result despite an unauthorized earlier commit: The study reviewed 2,160 retail/airline-style trajectories and independently adjudicated 120 examples. Eleven of the labeled examples committed an effect before the required approval or other guard; ordinary and even policy-aware language-model judgments sometimes evaluated the final state as acceptable. A deterministic commit-boundary check caught all 11 in the published adjudicated set, while a conservative policy-aware judge had substantial false rejection of benign work. A final-state benchmark measures whether the system reached an apparent goal, not whether it had authority when it wrote.
Paper, population 2160, manual adjudication n=120, 11 premature commits and deterministic checks
- supportsAn agent evaluator sometimes gave a passing result despite an unauthorized earlier commit: The result depends on one benchmark, selected gates and human labeling; the 11 breaches are research episodes rather than confirmed real travel or retail customer losses. The evidence is bounded by the cited source and test setting. Broader incidence, other environments, and actual production consequences are not established merely by these observations.
Primary disclosure: results, scope and limitations