Datadog compared one CodeGuard-assisted generation per model with earlier baseline builds of the same document portal. The experiment used CodeGuard 1.4.0 with Sonnet 5, Composer 2.5 and GPT 5.5. Method and findings. Mechanism, preconditions and effects. The tested instruction set requested security invariants, but the generated application still required…
Inspect the ClaimsCase · DiggingBeagle record
Security rules did not consistently become effective application controls
Single-run application comparison, not a representative model ranking.
Assessment profile
Assessment method- Setting
- evaluation
- Exploitation
- demonstrated
- Evidence
- Grade B
- Remediation
- not applicable
- AI role
- BY AI
Basis and provenance
CLM-CODEGUARD-CODING-RUNS-1 reports: The ownership IDOR persisted for Sonnet and Composer but was absent in the GPT security build; Composer's secret-check ordering left a fallback-secret guard ineffective. Source SRC-2026-CP17-DATADOG-CODEGUARD is located at 'Models using security rules; Analysis: Security rules versus no security rules'. The distinctive mechanism begins: The tested instruction set requested security invariants, but the generated application still required path-specific checks: a document owner must be compared with the authenticated principal at the read/write endpoint, and a fallback secret must be rejected before it is substituted. Classification: setting=evaluation, exploit_status=demonstrated, evidence_grade=B on the bounded original investigator/vendor or paper evidence, without independent dynamic replication. remediation_status=not_applicable; scoped control or limit: For this class of defect, test the unauthorized user's request at the actual API boundary and verify secret fallback behavior in the executing branch; simply detecting a guard in generated text does not prove the invariant. The null impact band avoids an ungrounded cross-Case severity rating; observed damage, targeted population and controlled test outcomes cannot be exchanged as the same quantity. Where a patch is described, its availability is separate from independently tested effectiveness.
Assessed Oct 23, 2026 using diggingbeagle.assessment/1.
Evidence basis: CLM-CODEGUARD-CODING-RUNS-1 · Datadog: Security rules in three coding-agent runs
At a glance
No independent reproduction; stochastic variation was not measured.
Limits and uncertaintyFull account
Datadog compared one CodeGuard-assisted generation per model with earlier baseline builds of the same document portal. The experiment used CodeGuard 1.4.0 with Sonnet 5, Composer 2.5 and GPT 5.5. Method and findings.
Mechanism, preconditions and effects. The tested instruction set requested security invariants, but the generated application still required path-specific checks: a document owner must be compared with the authenticated principal at the read/write endpoint, and a fallback secret must be rejected before it is substituted. In the single Datadog run, the Composer code made the fallback-secret check ineffective by executing it after defaulting; the ownership check remained defective in the Sonnet and Composer builds. These observations test resulting application behavior rather than the mere presence of security-looking code.
Comparison and evidence limits. The comparison cannot rank model families or estimate their vulnerability rates because there was one generation per configuration and no quantified run-to-run variance. It illustrates a security-rule implementation gap, not model refusal, and differs from scanner selection omissions, where a file never reached the reviewer at all.
Timeline
- Oct 19, 2026disclosure
Public disclosure
Claims & evidence
CLM-CODEGUARD-CODING-RUNS-1The ownership IDOR persisted for Sonnet and Composer but was absent in the GPT security build; Composer's secret-check ordering left a fallback-secret guard ineffective.supported
Basis: reported finding
- supportsDatadog: Security rules in three coding-agent runsprimary disclosure
Models using security rules; Analysis: Security rules versus no security rules
Controls and mitigations
- For this class of defect, test the unauthorized user's request at the actual API boundary and verify secret fallback behavior in the executing branch; simply detecting a guard in generated text does not prove the invariant.
Unknowns and contradictions
- No independent reproduction; stochastic variation was not measured.
Sources and citation
Material revision history
- Oct 23, 2026 · Canonical change recorded · new in release · revision 183