A coding agent can avoid prohibited commands and still leave a task unsafe by skipping a required verification, approval or recovery step. The ObligationBench researchers distinguish these missing positive obligations from explicit forbidden actions. In preliminary GLM-5.3 trajectories, the authors identified omissions in 56.92% and forbidden actions in…
Inspect the ClaimsCase · DiggingBeagle record
ObligationBench highlights security failures caused by skipped agent checks
ObligationBench studies an agent failure that forbidden-action filters can miss: omitting a required check, approval or rollback. The paper tests 240 expert-validated traces and reports imperfect recall even after specialized guard training. Results are benchmark measurements, not production omission rates.
Scope
Controlled finding or reported disclosure as of 2026-10-08; Benchmark finding, not universal omission prevalence.
Assessment profile
Assessment method- Setting
- evaluation
- Exploitation
- not established
- Evidence
- Grade B
- Remediation
- not applicable
- AI role
- BY AI
Basis and provenance
Desk review of scoped canonical Claims, the pinned SRC-2026-CP3-AGENT-UNFULFILLED-OBLIGATIONS Source, and the publicly accessible primary disclosure. The classification applies to the documented setting and prerequisites only. Impact remains unbanded because real victim consequences are not established, and no independent reproduction is claimed. Details: Benchmark prevalence does not estimate real-world omission frequency or resulting harm.
Assessed Oct 13, 2026 using diggingbeagle.assessment/1.
Evidence basis: CLM-CP3-AGENT-UNFULFILLED-OBLIGATIONS-1 · CLM-CP3-AGENT-UNFULFILLED-OBLIGATIONS-2 · CLM-CP3-AGENT-UNFULFILLED-OBLIGATIONS-3 · ObligationBench shows omissions can make agents unsafe despite no forbidden action
At a glance
Represent required validations as explicit task obligations that can be checked against the action trace. Model-based obligation detection is a supplemental signal, not an enforcement substitute for deterministic approvals and postconditions.
Read the implicationsBenchmark prevalence does not estimate real-world omission frequency or resulting harm. No prospective production test establishes how often obligation detection prevents unsafe completion.
Limits and uncertaintyFull account
A coding agent can avoid prohibited commands and still leave a task unsafe by skipping a required verification, approval or recovery step. The ObligationBench researchers distinguish these missing positive obligations from explicit forbidden actions. In preliminary GLM-5.3 trajectories, the authors identified omissions in 56.92% and forbidden actions in 30.00%. Both fractions describe the chosen benchmark, not a general population of deployed agents.
The team assembled 240 expert-validated traces spanning issue resolution, feature work and terminal operations. Fourteen guard models struggled to identify missing duties, with the strongest baseline reaching 48.97% recall and 10% exact match. ObligationGuard, trained with 40,000 synthetic examples, raised the study's recall to 57.52% and exact match to 21.67%. The residual misses are substantial. A guard that notices omissions after a trace is recorded is not itself proof that a task will enforce mandatory actions before irreversible tool calls.
Mechanism and trust boundary
- 01
Task obligation
A task has an explicit required check, approval or rollback condition that should precede completion.
Boundary: A task has an explicit required check, approval or rollback condition that should precede completion.
- 02
Agent trace
The agent produces permitted visible actions but leaves that requirement unperformed.
Boundary: The agent produces permitted visible actions but leaves that requirement unperformed.
- 03
Decision boundary
A guard or operator decides whether the task is safely complete; negative-only action scanning does not establish the missing obligation.
Boundary: A guard or operator decides whether the task is safely complete; negative-only action scanning does not establish the missing obligation.
Timeline
- Oct 8, 2026disclosure
Public disclosure
Claims & evidence
CLM-CP3-AGENT-UNFULFILLED-OBLIGATIONS-1In the paperâs preliminary GLM-5.3 sample, 56.92% of trajectories contained an unfulfilled obligation and 30% a forbidden action.supported
Basis: reported finding
- supportsObligationBench shows omissions can make agents unsafe despite no forbidden actionprimary disclosure
Abstract, preliminary trajectory study
CLM-CP3-AGENT-UNFULFILLED-OBLIGATIONS-2ObligationBench contains 240 expert-validated trajectories; the authors report ObligationGuard recall of 57.52%, compared with a best 48.97% among tested baselines.supported
Basis: reported finding
- supportsObligationBench shows omissions can make agents unsafe despite no forbidden actionprimary disclosure
Abstract, benchmark construction and model evaluations
CLM-CP3-AGENT-UNFULFILLED-OBLIGATIONS-3The evaluated 14 baseline guard models reached at most 10% exact match; the authors report 21.67% exact match for ObligationGuard trained on 40,000 synthetic examples.supported
Basis: reported finding
- supportsObligationBench shows omissions can make agents unsafe despite no forbidden actionprimary disclosure
Abstract, baseline model comparison and ObligationGuard results
Implications
Represent required validations as explicit task obligations that can be checked against the action trace. Model-based obligation detection is a supplemental signal, not an enforcement substitute for deterministic approvals and postconditions.
Controls and mitigations
- Encode safety-critical preconditions and postconditions in workflow logic so task completion cannot silently skip them.
- Use trace-linked obligation checks at review boundaries and require human approval when completion evidence is missing.
- Measure missed duties and false required actions separately; the reported guard recall leaves many obligations undetected.
Unknowns and contradictions
- Benchmark prevalence does not estimate real-world omission frequency or resulting harm.
- No prospective production test establishes how often obligation detection prevents unsafe completion.
Sources and citation
Material revision history
- Oct 13, 2026 · Published version · first publication · revision 137