Repository retrieval improved working code more than secure code in CASE-Bench

A coding assistant that retrieves more repository context may find the correct function signature or expected behavior while missing the defensive invariant that makes the patch safe. CASE-Bench reconstructs CVE tasks with both functional tests and exploit checks, so it can distinguish a bug fix that passes ordinary tests from one that closes the exploit path.

The 145 selected tasks and six model or harness conditions are a research sample rather than an estimate of real deployed coding-agent vulnerabilities. The upper-end gains describe tested configurations, not guaranteed improvements from any retrieval system.

The evaluator ran functional checks and exploit-sensitive tests on 145 CVE-derived tasks, so it could distinguish working patches from patches still vulnerable to the original flaw. Several ordinary retrieval configurations produced large correct-but-exploitable shares, while agents inspected relevant repository safeguards in only 27% to 36% of traces. A coding benchmark that records only whether tests passed would erase this difference. Security-aware context and an exploit regression check answer complementary questions, rather than making more context automatically safer.