Entity · DiggingBeagle record
SUSVIBES
Feature-request benchmark derived from real development changes that introduced vulnerabilities.
Record description
SUSVIBES evaluates coding agents on feature requests reconstructed from open-source development where vulnerable implementations were committed. It asks whether an agent can satisfy the requested behavior without reproducing a security failure. The published ICML 2026 study reports functional correctness and security outcomes separately.
This makes it a research instrument for security-sensitive code changes, not a count of compromised applications. Its task selection is deliberately vulnerability-focused, which limits extrapolation to ordinary development. The benchmark is also distinct from SecureVibe, the later training research that evaluates on it.