In simulated requests, a changed idempotency key could make an operation repeat after a timeout even when the original result was unknown. With a supplied new key, the benchmark recorded 1/80, 55/80 and 60/80 unsupported retries across its three model configurations. This is a controlled decision test, not a record of duplicate payments. A timeout gives…
Inspect the ClaimsCase · DiggingBeagle record
RetryBench catches unauthorized retries after ambiguous tool timeouts
Agent models differed substantially in whether they retried a possibly committed action with a fresh idempotency key.
Scope
Research claim attributed to the cited report, with limitations from intake DB-RADAR-CP7-CP7A-008.
Assessment profile
Assessment method- Setting
- simulation
- Exploitation
- demonstrated
- Evidence
- Grade C
- Remediation
- mitigation available
- AI role
- AGAINST AI
Basis and provenance
The simulated tool-timeout experiment counted unauthorized fresh-key retries, not completed duplicate payments. Reported 80-trial denominators apply only to tested model configurations. Assessment uses the pinned Claims and source identities; public primary pages were not independently retrievable during this run. Impact remains unrated because field consequence or a comparable scoring rubric is not established.
Assessed Oct 15, 2026 using diggingbeagle.assessment/1.
Evidence basis: CLM-CP7A-RETRYBENCH · RetryBench finds model-dependent unauthorized retries after ambiguous tool timeouts — original research or disclosure
At a glance
Independent reproduction or full primary-document access in this research run was not achieved. Real-world incidence and broader model or deployment generalization are not established unless specifically stated.
Limits and uncertaintyFull account
In simulated requests, a changed idempotency key could make an operation repeat after a timeout even when the original result was unknown. With a supplied new key, the benchmark recorded 1/80, 55/80 and 60/80 unsupported retries across its three model configurations. This is a controlled decision test, not a record of duplicate payments.
A timeout gives the agent uncertainty about completion, not permission to create a new operation identity. Target-side idempotency and read-after-write reconciliation should be evaluated independently of whether the model chooses to retry.
Claims & evidence
CLM-CP7A-RETRYBENCHThe cited authors or maintainers report the following bounded finding: Agent models differed substantially in whether they retried a possibly committed action with a fresh idempotency key.supported
Basis: reported finding
- supportsRetryBench finds model-dependent unauthorized retries after ambiguous tool timeouts — original research or disclosureprimary disclosure
Original primary document, reported finding and evaluation context; checkpoint DB-RADAR-CP7-CP7A-008
Unknowns and contradictions
- Independent reproduction or full primary-document access in this research run was not achieved.
- Real-world incidence and broader model or deployment generalization are not established unless specifically stated.
Sources and citation
Material revision history
- Oct 15, 2026 · Canonical change recorded · new in release · revision 150