News · DiggingBeagle record

Snyk's 108-flaw study found complementary strengths in static and AI code review

Across 20 curated applications, Snyk Code matched 89 known flaws and a Claude Code review matched 85; their union reached 101. Review-harness policy changed what the underlying model was allowed to report.

Published
Nov 1, 2026

The report

A code review can miss a real defect because the scanner lacks a rule, because the model overlooks the relevant data flow, or because the surrounding review workflow suppresses the finding. VulnBench 2.0 evaluated these possibilities against 108 curated vulnerabilities in 20 applications. Snyk Code matched 89, while a Claude Code configuration using Opus 5 at high effort matched 85. Together they matched 101, more than either one independently.

Those numbers describe the study's reference set and scoring procedure. They are not production detection rates, and their difference does not establish a general winner. The authors also showed that a review harness could prevent certain vulnerability classes from appearing in the final output even when changing the review instructions allowed the same model to identify more of them. Reporting policy is therefore a measured part of the security system, not merely a presentation preference.

Where the second reviewer helped

The useful result is the overlap analysis: static and model-based processes captured some different defects. Running another agent with an almost identical prompt would not necessarily reproduce that benefit. Teams need to compare distinct source-to-sink paths and verify any critical finding through execution or code tracing, rather than infer completeness from the number of models consulted.

The Case records the fixed benchmark conditions and the measurement limits. A policy change that makes an AI reviewer silent can appear to improve precision while removing true positives, so an audit should retain both the detector's internal observations and the final published finding list.

Research behind this