RegretBench reconstructs 205 reported regret moments as executable tasks. Affected state is deliberately placed outside native harness backup coverage, so the measured outcome concerns that boundary. Mechanism, preconditions and effects. RegretBench builds 205 executable coding-agent tasks around changes for which users later wished that prior state had…
Inspect the ClaimsCase · DiggingBeagle record
Completing an agent task does not establish that its effects can be reversed
Assessment profile
Assessment method- Setting
- evaluation
- Exploitation
- demonstrated
- Evidence
- Grade B
- Remediation
- mitigation available
- AI role
- WITH AI
Basis and provenance
CLM-CP17-REGRETBENCH-BASELINE reports: Across six evaluated model/harness configurations, task completion averaged 86.1% but backup before regret moments averaged 15.2%. Source SRC-2026-CP17-REGRETBENCH is located at 'Abstract; Sections 3-4'. The distinctive mechanism begins: RegretBench builds 205 executable coding-agent tasks around changes for which users later wished that prior state had been preserved. Classification: setting=evaluation, exploit_status=demonstrated, evidence_grade=B on the bounded original investigator/vendor or paper evidence, without independent dynamic replication. remediation_status=mitigation_available; scoped control or limit: Intercept state-changing tool calls before execution, capture preimages or snapshots with stable identities, and rehearse restoration against a known baseline. The published 92.2% score is an aggregate similarity measure and leaves incomplete recoveries unresolved. The null impact band avoids an ungrounded cross-Case severity rating; observed damage, targeted population and controlled test outcomes cannot be exchanged as the same quantity. Where a patch is described, its availability is separate from independently tested effectiveness.
Assessed Oct 23, 2026 using diggingbeagle.assessment/1.
Evidence basis: CLM-CP17-REGRETBENCH-BASELINE · CLM-CP17-REGRETBENCH-RECOVERY · Back Up Before You Regret: RegretBench and BackUpper
At a glance
Benchmark construction and similarity scoring limit generalization; no independent reproduction.
Limits and uncertaintyFull account
RegretBench reconstructs 205 reported regret moments as executable tasks. Affected state is deliberately placed outside native harness backup coverage, so the measured outcome concerns that boundary.
Mechanism, preconditions and effects. RegretBench builds 205 executable coding-agent tasks around changes for which users later wished that prior state had been preserved. The six tested models completed 86.1% of the tasks on average but made a useful backup before only 15.2% of regret moments. BackUpper wraps tool calls, captures potentially changed state using rules or a subagent, and reports 92.2% mean restoration similarity, not byte-for-byte perfect rollback.
Comparison and evidence limits. Task completion, backup decision and recovery quality are three different outcomes with different denominators. A backup mechanism can preserve more states than the user wants, while a selective mechanism can miss unusual side effects. This complements the effect-bound approval concern but focuses on reversibility after a permitted action, not authorization for the action itself.
Claims & evidence
CLM-CP17-REGRETBENCH-BASELINEAcross six evaluated model/harness configurations, task completion averaged 86.1% but backup before regret moments averaged 15.2%.supported
Basis: reported finding
- supportsBack Up Before You Regret: RegretBench and BackUpperprimary disclosure
Abstract; Sections 3-4
CLM-CP17-REGRETBENCH-RECOVERYBackUpper covered all tested regret-moment actions and achieved 92.2% average character-level similarity after restoration, not 92.2% exact recovery.supported
Basis: reported finding
- supportsBack Up Before You Regret: RegretBench and BackUpperprimary disclosure
Section 5, backup coverage and restoration fidelity; Appendix C.5
Controls and mitigations
- Intercept state-changing tool calls before execution, capture preimages or snapshots with stable identities, and rehearse restoration against a known baseline. The published 92.2% score is an aggregate similarity measure and leaves incomplete recoveries unresolved.
Unknowns and contradictions
- Benchmark construction and similarity scoring limit generalization; no independent reproduction.
Sources and citation
Material revision history
- Oct 23, 2026 · Canonical change recorded · new in release · revision 183