Case · DiggingBeagle record

A binary-only agent recovered many patched vulnerabilities but converted far fewer into working exploits

CyberBinGym gives agents vulnerable and patched binaries without source code or CVE descriptions. Researchers require a crash confined to the vulnerable binary to demonstrate a found issue, while a separate flag-capture or target-specific objective verifies a working exploit.

Public disclosure: Nov 7, 2026

Scope

An offensive test agent obtains binary artifacts for older vulnerable versions and their patched counterparts.

UnratedAI role: BY AI / WITH AIAssessment method

At a glance

Mechanism and trust boundary

  1. 01

    Starting condition

    An offensive test agent obtains binary artifacts for older vulnerable versions and their patched counterparts.

    Boundary: precondition and attacker capability

  2. 02

    Trust boundary crossed or evaluated

    The strongest reported model found 88 of 100 vulnerabilities but completed only 20 exploit objectives. On matched tasks, access to the patched counterpart sometimes helped exploit construction, yet reliable impact remained harder than vulnerability recognition.

    Boundary: mechanism

  3. 03

    Observed consequence

    The crucial boundary is from plausible vulnerability report to independently executed exploit effect; a crash alone is not an unauthorized production action.

    Boundary: observed effect and limitations

Claims & evidence

CLM-NOV15-CYBERBIN-MECHCyberBinGym gives agents vulnerable and patched binaries without source code or CVE descriptions. Researchers require a crash confined to the vulnerable binary to demonstrate a found issue, while a separate flag-capture or target-specific objective verifies a working exploit. The strongest reported model found 88 of 100 vulnerabilities but completed only 20 exploit objectives. On matched tasks, access to the patched counterpart sometimes helped exploit construction, yet reliable impact remained harder than vulnerability recognition.supported

Basis: reported finding

Link to claim
CLM-NOV15-CYBERBIN-SCOPEAll tasks use historical vulnerable releases under a controlled harness. The study is not an enumeration of 100 undisclosed zero-days, proof of live intrusions or evidence that every binary diff exposes a feasible exploit. The evidence is bounded by the cited source and test setting. Broader incidence, other environments, and actual production consequences are not established merely by these observations.supported

Basis: reported finding

Link to claim

Implications

The crucial boundary is from plausible vulnerability report to independently executed exploit effect; a crash alone is not an unauthorized production action.

Controls and mitigations

  • Require reproducible target-specific effects and compare against patched builds before accepting AI exploit claims.
  • Measure real containment and tool permissions separately from model ability; block test harness credentials and live egress outside authorized targets.

Unknowns and contradictions

  • The evidence is bounded by the cited source and test setting. Broader incidence, other environments, and actual production consequences are not established merely by these observations.

Sources and citation

Material revision history

  1. Nov 15, 2026 · Canonical change recorded · new in release · revision 250