Four places an AI security review can look complete and still miss the defect

A coding agent can report that it examined a repository, added a security check or repaired a bug. None of those statements alone establishes what code was inspected, whether the check governs the vulnerable request, or whether the patch preserved the rest of the application. Five October studies locate failures at different stages of that workflow and suggest distinct forms of executable evidence.

The scanner-omission study cataloged thirteen mechanisms across twelve selected vulnerability-discovery harnesses that could exclude a file before a model saw it. In a focused six-harness component test, moving a vulnerable file into a lower-visibility location excluded it for all six; five exclusions were silent. This does not establish that a model would have found the defect if it had seen the file. It establishes a prior failure: the system could not demonstrate review coverage.

A credible scan therefore records file discovery and exclusions separately from findings. An empty finding set without a coverage record is not equivalent to verified absence of vulnerabilities.

Datadog's CodeGuard comparison produced a different failure. Here the model did generate the application, but an owner check remained ineffective in two of the three particular runs, and one implementation assigned a fallback secret before checking whether a secret was missing. The test was one run per model/configuration, not a statistical ranking of coding models.

This is a security-rule implementation gap: a check can appear in source while failing on the relevant branch. Proof requires an unauthorized request or absent-secret scenario executed against the actual endpoint, not an explanation that the source contains a guard.

Trail of Bits retrospectively reviewed more than 130 merged security-fix pull requests and confirmed 21 regressions: sixteen functional, one performance and four build/test. None of those confirmed regressions was reported to break a security protection. One search pass generated 58 candidate regressions for the 21 confirmations, showing why flagged diffs require adjudication. The reported 25% false-positive reduction for its preliminary regression-hunt comparison has its own cohort and uncertainty; it is not an across-industry patch failure rate.

RegretBench tests a different post-action question. Its six configurations completed 86.1% of tasks on average but backed up before only 15.2% of regret moments. BackUpper reported 92.2% mean restoration similarity, explicitly not exact byte-for-byte rollback. Correctly repairing one feature and being able to undo unintended effects are separate obligations.

The nokast-secureRAG evaluation reported 99.4% recall on its internal instruction-injection distribution and identified only two of 300 external factual-poisoning passages. It would be inaccurate to call the 298 missed passages 298 successful downstream compromises: the study measured detector labels, not all resulting model answers. A security pipeline that recognizes explicit instructions may still admit plausible false facts.

These failures cannot be repaired by one overall "AI audit accuracy" score. Review coverage, executed invariants, regression fidelity, rollback and out-of-distribution detection measure different boundaries. Logging each one, with its own denominator, makes a clean report interpretable instead of reassuring by omission.