A.S.E. 2.0: repository context and secure code

Evidence record

CASE-Bench 145 real-CVE coding tasks and executable tests separating functional from secure outcomes.

Read the original source ↗

Claim-level citations (3)

• supportsRepository retrieval improved working code more than secure code in CASE-Bench: The CASE-Bench authors report that the tested coding agents encountered relevant repository safeguards in only 27% to 36% of their trajectories.

  Abstract; Results RQ3: Do Coding Agents Gather Security-Relevant Context?

• supportsRepository retrieval improved working code more than secure code in CASE-Bench: Across CASE-Bench conditions, functionality-oriented context increased functional pass rate by up to 8.2 percentage points but secure pass rate by at most 3.7 points.

  Abstract; Results RQ1 and context-construction comparisons

• supportsRepository retrieval improved working code more than secure code in CASE-Bench: Removing a relevant repository safeguard reduced secure passes by up to 8.1 points, and adding one insecure example raised functional-but-exploitable outcomes by 6.3 points.

  Abstract; Results RQ2 and context intervention