The researchers constructed 450 executable tasks across 150 scenarios, 24 CWE categories and seven languages. Functional completion ranged from 73.6% to 91.3%, but joint functionality-and-security success reached only 16.4% to 25.1%. The curated vulnerable workload cannot establish vulnerability prevalence in ordinary production work. A solution that…
Inspect the ClaimsCase · DiggingBeagle record
Secure Code Benchmark separates working agent code from security-correct code
A 450-task study reports much higher functional completion than joint functionality and security success across six coding agents.
Scope
Research claim attributed to the cited report, with limitations from intake DB-RADAR-CP7-CP7A-002.
Assessment profile
Assessment method- Setting
- evaluation
- Exploitation
- not established
- Evidence
- Grade C
- Remediation
- not applicable
- AI role
- AGAINST AI
Basis and provenance
The researchers scored curated coding tasks for both intended functionality and explicit security conditions. No claim about deployed vulnerable-code prevalence follows. Assessment uses the pinned Claims and source identities; public primary pages were not independently retrievable during this run. Impact remains unrated because field consequence or a comparable scoring rubric is not established.
Assessed Oct 15, 2026 using diggingbeagle.assessment/1.
Evidence basis: CLM-CP7A-SECURECODE · Last Secure Code Benchmark finds large functionality-security gap in AI coding agents — original research or disclosure
At a glance
Independent reproduction or full primary-document access in this research run was not achieved. Real-world incidence and broader model or deployment generalization are not established unless specifically stated.
Limits and uncertaintyFull account
The researchers constructed 450 executable tasks across 150 scenarios, 24 CWE categories and seven languages. Functional completion ranged from 73.6% to 91.3%, but joint functionality-and-security success reached only 16.4% to 25.1%. The curated vulnerable workload cannot establish vulnerability prevalence in ordinary production work.
A solution that passes functional tests can still fail the security condition specified by the task. Comparing only code-acceptance rates conceals that gap; the benchmark must preserve the same CWE and task populations across models.
Claims & evidence
CLM-CP7A-SECURECODEThe cited authors or maintainers report the following bounded finding: A 450-task study reports much higher functional completion than joint functionality and security success across six coding agents.supported
Basis: reported finding
- supportsLast Secure Code Benchmark finds large functionality-security gap in AI coding agents — original research or disclosureprimary disclosure
Original primary document, reported finding and evaluation context; checkpoint DB-RADAR-CP7-CP7A-002
Unknowns and contradictions
- Independent reproduction or full primary-document access in this research run was not achieved.
- Real-world incidence and broader model or deployment generalization are not established unless specifically stated.
Sources and citation
Material revision history
- Oct 15, 2026 · Canonical change recorded · new in release · revision 150