An auditor can be given internally consistent observations and still lack the fact that determines whether a deployed change is safe. The paper constructs matched lifecycles that are indistinguishable to a constrained verifier although their private outcomes differ. primary evidence makes the distinction actionable through deploy, hold, cannot determine and…
Inspect the ClaimsCase · DiggingBeagle record
Offline agent audits could not identify opposite later deployment outcomes
Post-Audit Mirage constructs safe and harmful lifecycles that expose identical verifier-visible traces, proving a limit on decisions made from those traces alone.
Assessment profile
Assessment method- Setting
- evaluation
- Exploitation
- not established
- Evidence
- Grade C
- Remediation
- not applicable
- AI role
- WITH AI
Basis and provenance
Conditional theoretical construction and experimental benchmark, not real deployment harm. The impact band remains unknown: available evidence does not establish actual deployed victim outcomes or a comparable measured loss.
Assessed Oct 26, 2026 using diggingbeagle.assessment/1.
Evidence basis: CLM-CP20-POSTAUDIT-MIRAGE · CLM-CP20-ENRICH-POSTAUDIT-MIRAGE · Post-Audit Mirage identical monitoring evidence can hide opposite agent outcomes
At a glance
The indistinguishability result is conditional on the audit observation model and the constructed environments, not proof that all offline agent evaluation is worthless.
Limits and uncertaintyFull account
An auditor can be given internally consistent observations and still lack the fact that determines whether a deployed change is safe. The paper constructs matched lifecycles that are indistinguishable to a constrained verifier although their private outcomes differ. primary evidence makes the distinction actionable through deploy, hold, cannot determine and unsupported decisions.
The indistinguishability result is conditional on the audit observation model and the constructed environments, not proof that all offline agent evaluation is worthless.
Add fresh outcome telemetry and explicit abstention; do not collapse a missing safety observation into a pass. Separate test-suite compliance from evidence that the real effect stayed authorized.
An offline verifier cannot distinguish two post-release worlds if every observation it receives is identical by construction. Post-Audit Mirage turns that identifiability limit into paired lifecycles and a monitor that may return deploy, hold, cannot determine or unsupported when fresh group-labelled deployment data are available. Its theorem is a conditional statement about the observation boundary, not proof that audits are useless. When a release has only code changes and test logs but lacks action-effect telemetry, the correct verdict on downstream safety can remain indeterminate. Require production observations before upgrading an offline pass into evidence of real-world outcomes.
Claims & evidence
CLM-CP20-POSTAUDIT-MIRAGEThe Post-Audit Mirage paper formalizes observational equivalence and an error/abstention tradeoff for post-deployment agent verification.supported
Basis: reported finding
- supportsPost-Audit Mirage identical monitoring evidence can hide opposite agent outcomesrole not specified
Abstract; indistinguishable lifecycle pairs; IdentifiedRangeMonitor
CLM-CP20-ENRICH-POSTAUDIT-MIRAGEPost-Audit Mirage constructs safe and harmful deployment lifecycles with identical offline verifier evidence; its IdentifiedRangeMonitor may abstain when post-deployment harm observations cannot identify the difference.supported
Basis: reported finding
- supportsPost-Audit Mirage identical monitoring evidence can hide opposite agent outcomesrole not specified
Abstract; observational-equivalence theorem and monitor output limitations
Unknowns and contradictions
- The indistinguishability result is conditional on the audit observation model and the constructed environments, not proof that all offline agent evaluation is worthless.
Sources and citation
Material revision history
- Oct 26, 2026 · Canonical change recorded · new in release · revision 193