The report
An agent transcript can become a repeatable software test if the expected result is made independently checkable. The open-source evals-from-sessions skill publishes a workflow for extracting bounded tasks from coding-agent histories, building a test environment and checking whether a model completed the work rather than merely sounding plausible. It is a development tool, not a newly reported security exploit.
From private work to a controlled benchmark
The author reports selecting ten automatically verifiable tasks from roughly 1,270 past sessions in Claude Code, Codex and Pi. The accompanying guide compares specific configurations on those tasks, with repeated runs, and limits its conclusions to the author's workload. The public repository describes six stages: copy and scan local session data, define task provenance, build a Harbor fixture, test its grader, run a smoke evaluation, and review failures before adding the task to a private collection.
The published process contains real safeguards. Its README specifies a Gitleaks scan of the copied transcripts, a separate task specification, human approval checkpoints and an explicit verifier check. A no-operation solution should fail, a known correct solution should pass, near misses should fail, and an alternative correct solution should succeed. The author also distinguishes an owner-controlled private repository snapshot from a synthetic, shareable task. Those checks make the proposal more concrete than simply replaying a conversation and calling it a benchmark.
Two distinct security questions
The first is data exposure. Coding sessions can contain repository paths, credentials, internal data and third-party material. An initial secret scan may miss information in generated fixtures, rewritten prompts, build layers, logs or grader output. The repository documents a procedure; its existence is not evidence that this project leaked secrets. Verification would use synthetic private markers and examine all outputs that leave the owner's environment, including model-provider requests.
The second is test integrity. A benchmark that includes future git history, reference patches or a grader the tested agent can edit may reward reproducing a previously successful trajectory instead of solving a new task. The repository addresses several of these leakage routes explicitly, but its guarantees still depend on implementation and the isolation of the verifier. ObligationBench illustrates how a completed-looking coding run can omit a required check. Transmissible reward hacking is a separate controlled example in which a grading shortcut can propagate through shared artifacts. Neither paper demonstrates a vulnerability in evals-from-sessions; they identify specific conditions a trustworthy session-derived eval should test.
Reusable private evals are valuable precisely when they measure externally verifiable states. A leaderboard score from ten personally selected tasks remains a comparison of those tasks and configurations, not a universal ranking of coding-agent quality or safety.