Evidence · DiggingBeagle record
CyberBinGym: Can AI Agents Turn Binary Patches into Real Exploits?
Benchmarks 100 historical binary vulnerabilities with patched-binary comparison and executable exploit verification.
- Published
- Nov 7, 2026
- Source role
- primary disclosure
Evidence record
Benchmarks 100 historical binary vulnerabilities with patched-binary comparison and executable exploit verification.
Claim-level citations (2)
- supportsA binary-only agent recovered many patched vulnerabilities but converted far fewer into working exploits: CyberBinGym gives agents vulnerable and patched binaries without source code or CVE descriptions. Researchers require a crash confined to the vulnerable binary to demonstrate a found issue, while a separate flag-capture or target-specific objective verifies a working exploit. The strongest reported model found 88 of 100 vulnerabilities but completed only 20 exploit objectives. On matched tasks, access to the patched counterpart sometimes helped exploit construction, yet reliable impact remained harder than vulnerability recognition.
CyberBinGym abstract and experimental results, 100 binary-only vulnerable/patched pairs, vulnerability vs exploit counts
- supportsA binary-only agent recovered many patched vulnerabilities but converted far fewer into working exploits: All tasks use historical vulnerable releases under a controlled harness. The study is not an enumeration of 100 undisclosed zero-days, proof of live intrusions or evidence that every binary diff exposes a feasible exploit. The evidence is bounded by the cited source and test setting. Broader incidence, other environments, and actual production consequences are not established merely by these observations.
Primary disclosure: results, scope and limitations