Ulysses verifies agent-generated web exploit claims against execution effects

An AI code auditor can produce a convincing exploit narrative without proving that it changed application state. Ulysses attempts to close that gap by repeatedly forming a vulnerability hypothesis, constructing an isolated test, observing the effect and revising the hypothesis when runtime evidence disagrees. Its benchmark includes real vulnerable application versions across multiple web stacks, so confirmation is tied to observed behavior rather than a second LLM agreeing with a description.

Ulysses synthesizes testable web exploit hypotheses and independently checks observable application effects; the authors report 10 successful proof-of-exploitability reproductions among 25 WebExploitGym advisories across five programming languages and seven frameworks. The study evaluates a selected 25-app reproducibility benchmark. Four additional proofs are not by themselves four new CVEs or evidence of real-world attacker compromises; they require separate validation and responsible disclosure. Verification coverage and execution costs are not established for arbitrary repositories.

Require isolated and authorized reproduction, a recorded before/after effect and explicit assessment of the exact vulnerable application version before publishing an exploitability claim. For agentic vulnerability review, runtime effect evidence is a stronger publication boundary than a model-produced description alone. Primary evidence.

Additional evidence and trust boundary

The verification boundary is observable application state, not the auditor's narrative confidence. Ulysses runs candidate payloads against isolated vulnerable web applications, checks the resulting effect and revises the hypothesis if evidence does not match. A proof in the local testbed may justify further disclosure, but it is not proof of live attack traffic or of a newly cataloged vulnerability identity. Supporting record.