The study analyzes failures around the model, including file selection and result handling. A clean output can therefore reflect omitted evidence rather than a successful vulnerability review. Mechanism, preconditions and effects. The paper examines file selection and intermediary analysis stages in twelve AI vulnerability-discovery harnesses and catalogs…
Inspect the ClaimsCase · DiggingBeagle record
Scanner selection rules can silently exclude vulnerable code before model review
Assessment profile
Assessment method- Setting
- evaluation
- Exploitation
- demonstrated
- Evidence
- Grade B
- Remediation
- mitigation available
- AI role
- AGAINST AI
Basis and provenance
CLM-CP17-SCANNER-OMISSIONS-MAP reports: The authors catalog 13 drop mechanisms across 12 selected open-source harnesses, with commit-specific source or instruction traces. Source SRC-2026-CP17-SCANNER-OMISSIONS is located at 'Section 1 contributions; Sections 3-4'. The distinctive mechanism begins: The paper examines file selection and intermediary analysis stages in twelve AI vulnerability-discovery harnesses and catalogs thirteen mechanisms that can keep code out of a model's review. Classification: setting=evaluation, exploit_status=demonstrated, evidence_grade=B on the bounded original investigator/vendor or paper evidence, without independent dynamic replication. remediation_status=mitigation_available; scoped control or limit: Report the exact scan universe and excluded files, track selection changes by diff, and test deliberate misplacement of known vulnerable examples. A dashboard that only counts findings cannot show code it never attempted to inspect. The null impact band avoids an ungrounded cross-Case severity rating; observed damage, targeted population and controlled test outcomes cannot be exchanged as the same quantity. Where a patch is described, its availability is separate from independently tested effectiveness.
Assessed Oct 23, 2026 using diggingbeagle.assessment/1.
Evidence basis: CLM-CP17-SCANNER-OMISSIONS-MAP · CLM-CP17-SCANNER-OMISSIONS-TEST · CLM-CP17-SCANNER-OMISSIONS-LIMIT · Where AI Vulnerability Discovery Harnesses Go Quiet
At a glance
These are analysis omissions, not demonstrated code execution or theft through scanner vulnerabilities.
Limits and uncertaintyFull account
The study analyzes failures around the model, including file selection and result handling. A clean output can therefore reflect omitted evidence rather than a successful vulnerability review.
Mechanism, preconditions and effects. The paper examines file selection and intermediary analysis stages in twelve AI vulnerability-discovery harnesses and catalogs thirteen mechanisms that can keep code out of a model's review. In its six-harness test, a vulnerable file moved to a lower-visibility location was excluded before inspection by every harness; five of those exclusions were silent. The test ran selection components independently and did not measure whether a later model with its own file tools would recover the omission.
Comparison and evidence limits. An omitted file is not proof the LLM would have detected the vulnerability had it been shown, and the twelve selected open-source projects are not a prevalence sample of all scanners. This is the inverse of the security-rule implementation gap: here the candidate code never enters the detector, rather than entering generation but failing a runtime invariant.
Claims & evidence
CLM-CP17-SCANNER-OMISSIONS-MAPThe authors catalog 13 drop mechanisms across 12 selected open-source harnesses, with commit-specific source or instruction traces.supported
Basis: reported finding
- supportsWhere AI Vulnerability Discovery Harnesses Go Quietprimary disclosure
Section 1 contributions; Sections 3-4
CLM-CP17-SCANNER-OMISSIONS-TESTIn the six-harness demonstration, five dropped the file silently before model inspection; the experiment did not test whether a model with file tools would recover it.supported
Basis: reported finding
- supportsWhere AI Vulnerability Discovery Harnesses Go Quietprimary disclosure
Section 7; Section 9.4; Section 10 limitations
CLM-CP17-SCANNER-OMISSIONS-LIMITThis illustrative sample is not a prevalence ranking. Authors examined their own OpenAnt and raptor most deeply and report several fixes before publication.supported
Basis: reported finding
- supportsWhere AI Vulnerability Discovery Harnesses Go Quietprimary disclosure
Sections 10-11
Controls and mitigations
- Report the exact scan universe and excluded files, track selection changes by diff, and test deliberate misplacement of known vulnerable examples. A dashboard that only counts findings cannot show code it never attempted to inspect.
Unknowns and contradictions
- These are analysis omissions, not demonstrated code execution or theft through scanner vulnerabilities.
Sources and citation
Material revision history
- Oct 23, 2026 · Canonical change recorded · new in release · revision 183