Researchers tested productized AI code-security flows and a deterministic static scanner against the same curated source-to-sink vulnerabilities across 20 applications. The leading AI review and static scanner each found defects the other missed, and their combined coverage exceeded either alone. A separate control showed that a review harness suppressed…
Inspect the ClaimsCase · DiggingBeagle record
VulnBench found AI review and deterministic SAST complementary, with harness-specific blind spots
In the 108-flaw VulnBench 2.0 study, Snyk Code matched 89 issues, Claude Code Opus 5 at high effort matched 85, and their union matched 101, while harness reporting policies excluded some real vulnerability classes.
Scope
108 curated findings in 20 open-source applications and fixed tool releases; no demonstrated field attack or general guarantee that adding more AI reviewers raises production security.
Assessment profile
Assessment method- Setting
- evaluation
- Exploitation
- demonstrated
- Evidence
- Grade B
- Remediation
- mitigation available
- AI role
- WITH AI
Basis and provenance
The study compares fixed scanning and agent review configurations against 108 curated flaws. The complementary union reflects this benchmark and its reporting constraints; it does not measure deployed code security improvements. Impact remains unrated because the evidence does not supply a calibrated scale for this Case.
Assessed Nov 1, 2026 using diggingbeagle.assessment/1.
Evidence basis: CLM-2026-NOV01-SNYK-VULNBENCH-HARNESS-1 · arXiv 2610.38617: deterministic SAST versus AI review in VulnBench 2.0
At a glance
Assess the reported tool or model action separately from the side effect actually observed.
Read the implications108 curated findings in 20 open-source applications and fixed tool releases; no demonstrated field attack or general guarantee that adding more AI reviewers raises production security.
Limits and uncertaintyFull account
Researchers tested productized AI code-security flows and a deterministic static scanner against the same curated source-to-sink vulnerabilities across 20 applications. The leading AI review and static scanner each found defects the other missed, and their combined coverage exceeded either alone. A separate control showed that a review harness suppressed vulnerability classes despite the model being capable of finding some of them under a different prompt. These are controlled scan outcomes rather than an estimate of vulnerabilities in every real codebase.
Mechanism and trust boundary
- 01
Step
Claims & evidence
CLM-2026-NOV01-SNYK-VULNBENCH-HARNESS-1In the 108-flaw VulnBench 2.0 study, Snyk Code matched 89 issues, Claude Code Opus 5 at high effort matched 85, and their union matched 101, while harness reporting policies excluded some real vulnerability classes.supported
Basis: reported finding
- supportsarXiv 2610.38617: deterministic SAST versus AI review in VulnBench 2.0primary disclosure
arXiv 2610.38617, Abstract, Main recall results, Harness policy and Threats to validity
Implications
Assess the reported tool or model action separately from the side effect actually observed.
Controls and mitigations
- Track source and sink locations, preserve scanner-specific recall and precision, and retain independent static or manual review instead of treating any one agent output as complete.
Unknowns and contradictions
- 108 curated findings in 20 open-source applications and fixed tool releases; no demonstrated field attack or general guarantee that adding more AI reviewers raises production security.
Sources and citation
Material revision history
- Nov 1, 2026 · Canonical change recorded · new in release · revision 211