A patch that suppresses one exploit can still violate maintainer functionality tests

Stopping a supplied proof-of-crash can be a poor proxy for repairing a vulnerability. In a Northwestern benchmark, agents attempted to patch 112 historical defects drawn from 84 repositories. The study ran both attack-oriented proof-of-concept tests and developer-authored functionality suites. A leading tested model suppressed around nine of ten original crash proofs, but passed the maintainer tests for only about six of ten tasks.

Those percentages have different denominators and meanings. The proof of concept asks whether one demonstration still triggers; the developer suite asks whether important application behavior survives. A patch could satisfy the first by disabling functionality entirely. Conversely, a legitimate functional patch might require environment setup absent from the benchmark. Results varied with agent scaffold and instructions, so model rankings without harness details are incomplete.

The defensible remediation judgment needs both an independent exploit test and regression coverage that represents maintainer intent. That also applies to AI-generated CodeQL fixes: clearing an alert is evidence that one scanner condition changed, not proof of exploit closure. The study describes a benchmark validation gap rather than a confirmed wave of broken production patches.