News · DiggingBeagle record

ToxicSkills audit finds scanner noise, unresolved candidates and detector gaps

A DiggingBeagle review of the intentionally malicious ToxicSkills demo separated reference-document false positives from two unresolved machine candidates and several detector-gap observations. The audit also records substantial coverage limits, so none of those gaps is promoted beyond the evidence actually preserved.

A dated report connected to the underlying research where available.

The report

The report

DiggingBeagle ran its deterministic Inspector against the exact pinned snapshot of `snyk-labs/toxicskills-goof`, a repository containing intentionally malicious agent-skill examples.

The inspection completed without tool errors and produced 19 correlated machine candidates. Human review did not treat those candidates as vulnerability verdicts. Seven were dismissed as false positives, ten were retained as informational, and two remained unresolved.

That split matters because several detections came from defensive reference material that deliberately contains examples of hostile instructions. A scanner that matches those strings without enough context can report the documentation of an attack as if it were the attack itself.

Two candidates remain unresolved

Two Snyk-skill candidates pointed toward concealed remote-installer or curl-to-shell behavior. The review considered them security-relevant, but did not promote either one to a confirmed or likely editorial finding.

The reason was provenance rather than absence of concern: the AuditReport v2 validator requires a material editorial finding to stay strictly inside the exact evidence range of its pinned machine finding. That exact range was not available in the returned audit package, so the review left both candidates unresolved instead of widening or inventing evidence.

The review also found detector gaps

Static review recorded three material observations that the correlated machine findings did not surface cleanly:

  • Vercel skill instructions involving host and `.env` data exfiltration;
  • an obfuscated network-to-shell prerequisite in the ClawHub skill;
  • a `UserPromptSubmit` hook in the Snyk skill that writes environment data.

These remain detector-gap observations, not machine-provenanced findings. The audit deliberately keeps that distinction visible.

Coverage was incomplete

The pinned acquisition contained 33 included files and 21 skipped files. One file had an extraction error, nine surfaces were skipped or reduced, Git history and some repository object types were outside the acquired snapshot, and the offline known-vulnerability database adapter was not applicable.

The result is therefore not a safety verdict on the repository. It is a bounded test of the inspection pipeline: acquisition and deterministic scanning worked, contextual false positives were exposed, security-relevant candidates were preserved without overstating them, and the review identified behaviors that the current detector set should learn to capture more reliably.

Research behind this

Cite this record

DiggingBeagle. “ToxicSkills audit finds scanner noise, unresolved candidates and detector gaps.” https://diggingbeagle.com/news/toxicskills-audit-scanner-noise-unresolved-candidates-detector-gaps/

Citation guidance

Independent research

The source stays with the story.

Claims, evidence and corrections remain inspectable. About the project · Our methodology