A local listwise coding agent inspects retrieved commit candidates to link CVEs to likely fixing changes, but selection accuracy does not establish that a patch is safe or complete. Hybrid retrieval yields a set of 100 candidate fixing commits; a frozen open-weight agent compares candidates within one conversation and inspects a small shortlist using…
Inspect the ClaimsCase · DiggingBeagle record
PatchHolmes improves fixing-commit ranking without proving patch correctness
A local listwise coding agent inspects retrieved commit candidates to link CVEs to likely fixing changes, but selection accuracy does not establish that a patch is safe or complete.
AACL-IJCNLP 2026 research comparison on supplied CVE/commit candidate pools, with selective tool calls and no proof of deployed mitigation.
Assessment profile
Assessment method- Setting
- evaluation
- Exploitation
- not established
- Evidence
- Grade B
- Remediation
- not applicable
- AI role
- WITH AI
Basis and provenance
Measured top-one retrieval improves on evaluated datasets, but no exploit prevention, deployment or fixing completeness follows from commit ranking alone. Classified as evaluation with exploit status not_established. Evidence grade B reflects the original paper or maintainer advisory and its stated limitations, not an independent audit of all deployments. Comparable realized harm is not established, so impact stays unrated; remediation status not_applicable is limited to the cited experiment or affected software version. New source-specific boundary: Reported commit-ranking efficiency should be separated from the number of commits actually inspected and from any subsequent reproduction of the repaired vulnerability.
Assessed Nov 15, 2026 using diggingbeagle.assessment/1.
Evidence basis: CLM-R245-PATCHHOLMES-FIX-LINKS-01 · CLM-R245-PATCHHOLMES-FIX-LINKS-02 · CLM-R246-PATCHHOLMES-FIX-LINKS-BOUNDARY · CLM-ENR250-08 · PatchHolmes improves fixing-commit ranking without proving patch correctness (research paper)
At a glance
Private or embargoed fixes and backports may be absent from public history, and a selected commit may still contain an incomplete or regressed fix. Reported retrieval gains cannot be treated as reduced exploitation. No evidence here establishes the number of production victims or a broader failure rate outside the cited evaluation.
Limits and uncertaintyFull account
A local listwise coding agent inspects retrieved commit candidates to link CVEs to likely fixing changes, but selection accuracy does not establish that a patch is safe or complete.
Hybrid retrieval yields a set of 100 candidate fixing commits; a frozen open-weight agent compares candidates within one conversation and inspects a small shortlist using bounded local tools. For a PatchFinder top-ten transfer set, the authors report top-one retrieval improving from 24.28 to 39.86 percent under their candidate and ground-truth labels.
Private or embargoed fixes and backports may be absent from public history, and a selected commit may still contain an incomplete or regressed fix. Reported retrieval gains cannot be treated as reduced exploitation.
The relevant security dependency is evidence linking, not automatically repairing code. Hybrid retrieval produces roughly 100 candidate commits per CVE. A local agent compares the candidate set together, then spends a limited tool budget inspecting the most promising few. This provides an evidence trail from a vulnerability record to a potential fixing commit, but candidate recall and labels still determine whether the correct answer can appear. A vendor backport or a private fix may be absent from every public repository.
The independent candidate-set result is more informative than a pooled headline gain: on PatchFinder_top10 the local listwise model improves top-one selection from 24.28 to 39.86 percent with the same supplied candidates. The improvement does not establish patch completeness, shipping status or real exploit prevention. A security claim should still compare the CVE conditions against code and package versions rather than treating a high-ranked Git hash as remediation evidence.
Mechanism and trust boundary
- 01
Entry or starting condition
A public CVE lacks a reliable direct link to the change that addressed it.
Boundary: attacker or research precondition
- 02
Crossing the relevant boundary
Hybrid retrieval offers candidate commits and a local listwise model selectively inspects repository context to choose a likely fix.
Boundary: mechanism and authorization boundary
- 03
Observed or reported result
Measured top-one retrieval improves on evaluated datasets, but no exploit prevention, deployment or fixing completeness follows from commit ranking alone.
Boundary: measured outcome and evidentiary limit
Claims & evidence
CLM-ENR250-08PatchHolmes first ranks a window of roughly 100 plausible repository commits and then inspects a much smaller selection, typically three to ten commits, through its constrained inspection tools. Ranking candidates is not the same event as verifying that a proposed commit repairs the vulnerable path.supported
Basis: reported finding
- supportsPatchHolmes improves fixing-commit ranking without proving patch correctness (research paper)primary disclosure
Approach and efficiency evaluation: initial candidate window, iterative inspection budget and final fix-link verification
CLM-R245-PATCHHOLMES-FIX-LINKS-01For a PatchFinder top-ten transfer set, the authors report top-one retrieval improving from 24.28 to 39.86 percent under their candidate and ground-truth labels.supported
Basis: reported finding
- supportsPatchHolmes improves fixing-commit ranking without proving patch correctness (research paper)primary disclosure
Results, PatchFinder transfer evaluation
CLM-R245-PATCHHOLMES-FIX-LINKS-02Hybrid retrieval yields a set of 100 candidate fixing commits; a frozen open-weight agent compares candidates within one conversation and inspects a small shortlist using bounded local tools.supported
Basis: reported finding
- supportsPatchHolmes improves fixing-commit ranking without proving patch correctness (research paper)primary disclosure
Approach, hybrid retrieval and listwise commit inspection
CLM-R246-PATCHHOLMES-FIX-LINKS-BOUNDARYThe authors report that moving PatchHolmes to the independent PatchFinder_top10 candidate set raises top-one fixing-commit retrieval from 24.28 percent to 39.86 percent; this is candidate identification accuracy, not proof that the linked commit securely fixes a deployed product.supported
Basis: reported finding
- supportsPatchHolmes improves fixing-commit ranking without proving patch correctness (research paper)primary disclosure
AACL 2026 paper 115, Abstract and cross-dataset transfer evaluation
Unknowns and contradictions
- Private or embargoed fixes and backports may be absent from public history, and a selected commit may still contain an incomplete or regressed fix. Reported retrieval gains cannot be treated as reduced exploitation.
- No evidence here establishes the number of production victims or a broader failure rate outside the cited evaluation.
Sources and citation
Material revision history
- Nov 15, 2026 · Canonical change recorded · new in release · revision 251