CyberBinGym gives agents vulnerable and patched binaries without source code or CVE descriptions. Researchers require a crash confined to the vulnerable binary to demonstrate a found issue, while a separate flag-capture or target-specific objective verifies a working exploit. The strongest reported model found 88 of 100 vulnerabilities but completed only…
Inspect the ClaimsCase · DiggingBeagle record
A binary-only agent recovered many patched vulnerabilities but converted far fewer into working exploits
CyberBinGym gives agents vulnerable and patched binaries without source code or CVE descriptions. Researchers require a crash confined to the vulnerable binary to demonstrate a found issue, while a separate flag-capture or target-specific objective verifies a working exploit.
Public disclosure: Nov 7, 2026
An offensive test agent obtains binary artifacts for older vulnerable versions and their patched counterparts.
At a glance
The crucial boundary is from plausible vulnerability report to independently executed exploit effect; a crash alone is not an unauthorized production action.
Read the implicationsThe evidence is bounded by the cited source and test setting. Broader incidence, other environments, and actual production consequences are not established merely by these observations.
Limits and uncertaintyFull account
CyberBinGym gives agents vulnerable and patched binaries without source code or CVE descriptions. Researchers require a crash confined to the vulnerable binary to demonstrate a found issue, while a separate flag-capture or target-specific objective verifies a working exploit.
The strongest reported model found 88 of 100 vulnerabilities but completed only 20 exploit objectives. On matched tasks, access to the patched counterpart sometimes helped exploit construction, yet reliable impact remained harder than vulnerability recognition.
All tasks use historical vulnerable releases under a controlled harness. The study is not an enumeration of 100 undisclosed zero-days, proof of live intrusions or evidence that every binary diff exposes a feasible exploit.
Reported figures and denominator. On 100 historical flaws, strongest reported agent recovered 88 and achieved 20 exploit objectives.
The evidence is bounded by the cited source and test setting. Broader incidence, other environments, and actual production consequences are not established merely by these observations.
Mechanism and trust boundary
- 01
Starting condition
An offensive test agent obtains binary artifacts for older vulnerable versions and their patched counterparts.
Boundary: precondition and attacker capability
- 02
Trust boundary crossed or evaluated
The strongest reported model found 88 of 100 vulnerabilities but completed only 20 exploit objectives. On matched tasks, access to the patched counterpart sometimes helped exploit construction, yet reliable impact remained harder than vulnerability recognition.
Boundary: mechanism
- 03
Observed consequence
The crucial boundary is from plausible vulnerability report to independently executed exploit effect; a crash alone is not an unauthorized production action.
Boundary: observed effect and limitations
Claims & evidence
CLM-NOV15-CYBERBIN-MECHCyberBinGym gives agents vulnerable and patched binaries without source code or CVE descriptions. Researchers require a crash confined to the vulnerable binary to demonstrate a found issue, while a separate flag-capture or target-specific objective verifies a working exploit. The strongest reported model found 88 of 100 vulnerabilities but completed only 20 exploit objectives. On matched tasks, access to the patched counterpart sometimes helped exploit construction, yet reliable impact remained harder than vulnerability recognition.supported
Basis: reported finding
- supportsCyberBinGym: Can AI Agents Turn Binary Patches into Real Exploits?primary disclosure
CyberBinGym abstract and experimental results, 100 binary-only vulnerable/patched pairs, vulnerability vs exploit counts
CLM-NOV15-CYBERBIN-SCOPEAll tasks use historical vulnerable releases under a controlled harness. The study is not an enumeration of 100 undisclosed zero-days, proof of live intrusions or evidence that every binary diff exposes a feasible exploit. The evidence is bounded by the cited source and test setting. Broader incidence, other environments, and actual production consequences are not established merely by these observations.supported
Basis: reported finding
- supportsCyberBinGym: Can AI Agents Turn Binary Patches into Real Exploits?primary disclosure
Primary disclosure: results, scope and limitations
Implications
The crucial boundary is from plausible vulnerability report to independently executed exploit effect; a crash alone is not an unauthorized production action.
Controls and mitigations
- Require reproducible target-specific effects and compare against patched builds before accepting AI exploit claims.
- Measure real containment and tool permissions separately from model ability; block test harness credentials and live egress outside authorized targets.
Unknowns and contradictions
- The evidence is bounded by the cited source and test setting. Broader incidence, other environments, and actual production consequences are not established merely by these observations.
Sources and citation
Material revision history
- Nov 15, 2026 · Canonical change recorded · new in release · revision 250