Case · DiggingBeagle record

PatchHolmes improves fixing-commit ranking without proving patch correctness

A local listwise coding agent inspects retrieved commit candidates to link CVEs to likely fixing changes, but selection accuracy does not establish that a patch is safe or complete.

Scope

AACL-IJCNLP 2026 research comparison on supplied CVE/commit candidate pools, with selective tool calls and no proof of deployed mitigation.

Assessment profile

Assessment method
Setting
evaluation
Exploitation
not established
Evidence
Grade B
Remediation
not applicable
AI role
WITH AI
Basis and provenance

Measured top-one retrieval improves on evaluated datasets, but no exploit prevention, deployment or fixing completeness follows from commit ranking alone. Classified as evaluation with exploit status not_established. Evidence grade B reflects the original paper or maintainer advisory and its stated limitations, not an independent audit of all deployments. Comparable realized harm is not established, so impact stays unrated; remediation status not_applicable is limited to the cited experiment or affected software version. New source-specific boundary: Reported commit-ranking efficiency should be separated from the number of commits actually inspected and from any subsequent reproduction of the repaired vulnerability.

Assessed Nov 15, 2026 using diggingbeagle.assessment/1.

At a glance

Mechanism and trust boundary

  1. 01

    Entry or starting condition

    A public CVE lacks a reliable direct link to the change that addressed it.

    Boundary: attacker or research precondition

  2. 02

    Crossing the relevant boundary

    Hybrid retrieval offers candidate commits and a local listwise model selectively inspects repository context to choose a likely fix.

    Boundary: mechanism and authorization boundary

  3. 03

    Observed or reported result

    Measured top-one retrieval improves on evaluated datasets, but no exploit prevention, deployment or fixing completeness follows from commit ranking alone.

    Boundary: measured outcome and evidentiary limit

Claims & evidence

CLM-ENR250-08PatchHolmes first ranks a window of roughly 100 plausible repository commits and then inspects a much smaller selection, typically three to ten commits, through its constrained inspection tools. Ranking candidates is not the same event as verifying that a proposed commit repairs the vulnerable path.supported

Basis: reported finding

Link to claim

Unknowns and contradictions

  • Private or embargoed fixes and backports may be absent from public history, and a selected commit may still contain an incomplete or regressed fix. Reported retrieval gains cannot be treated as reduced exploitation.
  • No evidence here establishes the number of production victims or a broader failure rate outside the cited evaluation.

Sources and citation

Material revision history

  1. Nov 15, 2026 · Canonical change recorded · new in release · revision 251