JevAct integrates policy questions and compiled validation into a model tool workflow. The study reports improvements in specific benchmark task success, but its retry semantics expose an independent security issue: rejection can become permission to try again until a later proposal is committed unchecked. A different user simulator also changes the…
Inspect the ClaimsCase · DiggingBeagle record
JevAct decision gates can fail open after denied agent writes
A decision gate improves some benchmark task success while a bounded retry rule can commit writes that failed authorization checks.
Scope
The results are from research benchmarks; exact deployed configuration, independent replication and impact outside the test are unknown.
Assessment profile
Assessment method- Setting
- simulation
- Exploitation
- demonstrated
- Evidence
- Grade C
- Remediation
- not applicable
- AI role
- AGAINST AI
Basis and provenance
The pinned Claim and its primary disclosure support the described controlled result, but do not establish a population-wide rate or calibrated impact. The source is primary to the described analysis, not independent replication of every outcome. Impact remains unbanded because no calibrated impact rubric for this particular evidence is established.
Assessed Nov 2, 2026 using diggingbeagle.assessment/1.
Evidence basis: CLM-2026-NOV02-JEVACT-FAILOPEN-RETRY-A · Think Before You Act: Decision Gate for LLM Agents
At a glance
Authorization failure must not turn into automatic permission simply because an agent repeats the request.
Read the implicationsThe results are from research benchmarks; exact deployed configuration, independent replication and impact outside the test are unknown. The results do not establish that the compiled checks cover changes in user intent or that a production backend independently refuses a write after retry exhaustion.
Limits and uncertaintyFull account
JevAct integrates policy questions and compiled validation into a model tool workflow. The study reports improvements in specific benchmark task success, but its retry semantics expose an independent security issue: rejection can become permission to try again until a later proposal is committed unchecked. A different user simulator also changes the measured gain. Neither benchmark success nor a compiled rule should be interpreted as proof that every state mutation is governed by that rule.
A pre-execution judge is useful only to the extent that every material write remains within the approval boundary. The paper reports strong task-completion results in some telecom settings, but also notes that 42 of 101 incorrect labelled development writes violated no compiled question: the checks encoded policy incompletely. A bounded revision loop should not silently turn repeated rejections into permission for a later unchecked write. The evaluation measures a specific agent harness and simulator; higher task pass rates are not an independent estimate of safe financial or administrative actions.
Mechanism and trust boundary
- 01
Propose a state-changing tool call
The model supplies parameters to a write gate.
Boundary: Agent intention into authorization predicate
- 02
Receive repeated denials
The retry loop counts failed proposals as part of task recovery.
Boundary: Policy refusal into control-flow state
- 03
Commit a later unchecked write
The reported fallback permits a mutation without the same verification.
Boundary: Retry limit into write permission
Claims & evidence
CLM-2026-NOV02-JEVACT-FAILOPEN-RETRY-AThe JevAct paper describes a decision-gating design that, after two denied attempts, can allow a subsequent proposed write without the same checks; the authors report 42 of 101 invalid writes passed a compiled question condition.supported
Basis: reported finding
- supportsThink Before You Act: Decision Gate for LLM Agentsprimary disclosure
Decision loop, Write gate and Results, limitations
Implications
Authorization failure must not turn into automatic permission simply because an agent repeats the request.
Controls and mitigations
- Fail closed on repeated denials, escalate to a separate approver and keep retry counts independent from action authorization.
- Treat exhausted verification attempts as a hard deny or explicit human escalation; test wrong-but-valid tool writes against independent business invariants.
Unknowns and contradictions
- The results are from research benchmarks; exact deployed configuration, independent replication and impact outside the test are unknown.
- The results do not establish that the compiled checks cover changes in user intent or that a production backend independently refuses a write after retry exhaustion.
Sources and citation
Material revision history
- Nov 2, 2026 · Canonical change recorded · new in release · revision 214