The study reviewed 2,160 retail/airline-style trajectories and independently adjudicated 120 examples. Eleven of the labeled examples committed an effect before the required approval or other guard; ordinary and even policy-aware language-model judgments sometimes evaluated the final state as acceptable. A deterministic commit-boundary check caught all 11…
Inspect the ClaimsCase · DiggingBeagle record
An agent evaluator sometimes gave a passing result despite an unauthorized earlier commit
The study reviewed 2,160 retail/airline-style trajectories and independently adjudicated 120 examples. Eleven of the labeled examples committed an effect before the required approval or other guard; ordinary and even policy-aware language-model judgments sometimes evaluated the final state as acceptable.
Public disclosure: Nov 10, 2026
A multi-step agent must change a real or simulated reservation/order and the benchmark scores the final outcome.
At a glance
The critical evidence is an approval receipt bound to the actor, resource and exact write before the transaction commits.
Read the implicationsThe evidence is bounded by the cited source and test setting. Broader incidence, other environments, and actual production consequences are not established merely by these observations.
Limits and uncertaintyFull account
The study reviewed 2,160 retail/airline-style trajectories and independently adjudicated 120 examples. Eleven of the labeled examples committed an effect before the required approval or other guard; ordinary and even policy-aware language-model judgments sometimes evaluated the final state as acceptable.
A deterministic commit-boundary check caught all 11 in the published adjudicated set, while a conservative policy-aware judge had substantial false rejection of benign work. A final-state benchmark measures whether the system reached an apparent goal, not whether it had authority when it wrote.
The result depends on one benchmark, selected gates and human labeling; the 11 breaches are research episodes rather than confirmed real travel or retail customer losses.
Reported figures and denominator. 2,160 trajectories; 120 independently adjudicated; 11 premature commitments in the labeled sample and 11/11 detected by deterministic boundary checks.
The evidence is bounded by the cited source and test setting. Broader incidence, other environments, and actual production consequences are not established merely by these observations.
Mechanism and trust boundary
- 01
Starting condition
A multi-step agent must change a real or simulated reservation/order and the benchmark scores the final outcome.
Boundary: precondition and attacker capability
- 02
Trust boundary crossed or evaluated
A deterministic commit-boundary check caught all 11 in the published adjudicated set, while a conservative policy-aware judge had substantial false rejection of benign work. A final-state benchmark measures whether the system reached an apparent goal, not whether it had authority when it wrote.
Boundary: mechanism
- 03
Observed consequence
The critical evidence is an approval receipt bound to the actor, resource and exact write before the transaction commits.
Boundary: observed effect and limitations
Claims & evidence
CLM-NOV15-JUDGE-MECHThe study reviewed 2,160 retail/airline-style trajectories and independently adjudicated 120 examples. Eleven of the labeled examples committed an effect before the required approval or other guard; ordinary and even policy-aware language-model judgments sometimes evaluated the final state as acceptable. A deterministic commit-boundary check caught all 11 in the published adjudicated set, while a conservative policy-aware judge had substantial false rejection of benign work. A final-state benchmark measures whether the system reached an apparent goal, not whether it had authority when it wrote.supported
Basis: reported finding
- supportsWhen Evaluators of AI Agents Disagreeprimary disclosure
Paper, population 2160, manual adjudication n=120, 11 premature commits and deterministic checks
CLM-NOV15-JUDGE-SCOPEThe result depends on one benchmark, selected gates and human labeling; the 11 breaches are research episodes rather than confirmed real travel or retail customer losses. The evidence is bounded by the cited source and test setting. Broader incidence, other environments, and actual production consequences are not established merely by these observations.supported
Basis: reported finding
- supportsWhen Evaluators of AI Agents Disagreeprimary disclosure
Primary disclosure: results, scope and limitations
Implications
The critical evidence is an approval receipt bound to the actor, resource and exact write before the transaction commits.
Controls and mitigations
- Record immutable, time-ordered authorization receipts at every mutating tool boundary rather than asking a judge to infer approval from the finished transcript.
- Measure unauthorized commits and legitimate-task false refusals separately; a judge that blocks all actions is not a useful secure executor.
Unknowns and contradictions
- The evidence is bounded by the cited source and test setting. Broader incidence, other environments, and actual production consequences are not established merely by these observations.
Sources and citation
Material revision history
- Nov 15, 2026 · Canonical change recorded · new in release · revision 250