Trail of Bits reviewed its merged Patch the Planet contributions using before/after reproductions and engineer review. Its new plugins organize patch creation and independent regression checking. Mechanism, preconditions and effects. Trail of Bits retrospectively inspected more than 130 merged security-fix pull requests with before-and-after executable…
Inspect the ClaimsCase · DiggingBeagle record
AI-assisted security repair still needs regression validation
Assessment profile
Assessment method- Setting
- evaluation
- Exploitation
- demonstrated
- Evidence
- Grade B
- Remediation
- mitigation available
- AI role
- WITH AI
Basis and provenance
CLM-CP17-PATCH-REGRESSIONS-COHORT reports: Review of over 130 merged PRs confirmed 21 regressions: 16 functional, one performance and four build/test; none broke a security protection. Source SRC-2026-CP17-PATCH-REGRESSIONS is located at 'Standardizing the patching process, first three paragraphs'. The distinctive mechanism begins: Trail of Bits retrospectively inspected more than 130 merged security-fix pull requests with before-and-after executable checks and human review. Classification: setting=evaluation, exploit_status=demonstrated, evidence_grade=B on the bounded original investigator/vendor or paper evidence, without independent dynamic replication. remediation_status=mitigation_available; scoped control or limit: Require runnable before-and-after reproduction for each suspected regression and keep the project's intended behavior and threat model in the check. The proposed plugins support review, not fully automatic proof that a patch is safe. The null impact band avoids an ungrounded cross-Case severity rating; observed damage, targeted population and controlled test outcomes cannot be exchanged as the same quantity. Where a patch is described, its availability is separate from independently tested effectiveness.
Assessed Oct 23, 2026 using diggingbeagle.assessment/1.
Evidence basis: CLM-CP17-PATCH-REGRESSIONS-COHORT · CLM-CP17-PATCH-REGRESSIONS-PRECISION · Trail of Bits: patch review and regression-hunt
At a glance
Selected project cohort and preliminary comparison are not general regression rates. No independent rerun.
Limits and uncertaintyFull account
Trail of Bits reviewed its merged Patch the Planet contributions using before/after reproductions and engineer review. Its new plugins organize patch creation and independent regression checking.
Mechanism, preconditions and effects. Trail of Bits retrospectively inspected more than 130 merged security-fix pull requests with before-and-after executable checks and human review. It confirmed 21 regressions: 16 functional, one performance and four build/test; none was a reported regression that broke a security protection. A single search pass generated 58 candidate regressions for those 21 confirmations, demonstrating that finding a suspicious diff is not the same as adjudicating new breakage.
Comparison and evidence limits. The preliminary comparison says regression-hunt produced 25% fewer false positives than a prompted baseline, with a 95% interval of 14% to 38%; stricter evidence gates also left some known regressions borderline. Neither statistic is a global rate for coding-agent patches. Compare with the Datadog CodeGuard application test: one asks whether a proposed security invariant works, the other asks whether a change broke previously supported behavior.
Timeline
- Oct 21, 2026disclosure
Public disclosure
Claims & evidence
CLM-CP17-PATCH-REGRESSIONS-COHORTReview of over 130 merged PRs confirmed 21 regressions: 16 functional, one performance and four build/test; none broke a security protection.supported
Basis: reported finding
- supportsTrail of Bits: patch review and regression-huntprimary disclosure
Standardizing the patching process, first three paragraphs
CLM-CP17-PATCH-REGRESSIONS-PRECISIONOne run produced 58 candidates for those 21 confirmations. Preliminary regression-hunt evaluation reported 25% fewer false positives than a baseline Codex review prompt.supported
Basis: reported finding
- supportsTrail of Bits: patch review and regression-huntprimary disclosure
Introductory evaluation paragraph; Standardizing the patching process
Controls and mitigations
- Require runnable before-and-after reproduction for each suspected regression and keep the project's intended behavior and threat model in the check. The proposed plugins support review, not fully automatic proof that a patch is safe.
Unknowns and contradictions
- Selected project cohort and preliminary comparison are not general regression rates. No independent rerun.
Sources and citation
Material revision history
- Oct 23, 2026 · Canonical change recorded · new in release · revision 183