Can Adversarial Code Comments Fool AI Security Reviewers: large-scale empirical study

Evidence record

Directly inspected source-code review study, arXiv:2602.16741v1, by Scott Thornton. Eight models were tested on 100 vulnerable Python, JavaScript and Java samples under eight comment variants, producing 9,366 primary analyses. Adversarial comments changed detection only modestly without statistical significance (McNemar exact p > 0.21), while cross-referenced static-analysis signals helped in a separate defense experiment. This is a contrasting task and representation, not independent replication or refutation of Schefold's synthetic EVM bytecode benchmark.

Read the original source ↗

Claim-level citations (1)

• supportsThe Deception Delta: synthetic fund-drain contracts mislead AI bytecode auditors: A separate study by Scott Thornton tested eight models on 100 vulnerable source-code samples across Python, JavaScript and Java, obtaining 9,366 primary evaluations. It reported small, statistically non-significant effects from adversarial comments (McNemar exact p > 0.21). This is a contrast, not an independent replication of Deception Delta: Thornton evaluates comments over visible conventional vulnerabilities, whereas Schefold evaluates end-to-end deceptive EVM contract bytecode without source or execution tools.

  arXiv abstract and HTML Sections 4 Methodology, 5.2 Adversarial Comments Produce Small, Non-Significant Effects; eight models, 100 samples, 9,366 primary evaluations, McNemar p > 0.21