Analysis · DiggingBeagle record

A scanner can say SAFE after it skipped the instruction, the dependency or the entire check

Three SkillSpector reports test distinct coverage failures: untranslated attack text, an invisible BOM, and a proxy error that stops dependency analysis. None is equivalent to proof of a clean skill.

Published
Nov 2, 2026

Overview

A scan result answers only what the scanner actually examined. Three October and November SkillSpector reports illustrate why dashboards need to separate the scope of inspection from the severity of what was found. Their technical triggers are different, and they should not be reported as one generalized bypass.

Language coverage: meaning is not an English keyword

In a test of SkillSpector 2.12.0 with --no-llm, a researcher provided four paired English and Chinese skill instructions. The English versions received warnings; three of the corresponding Chinese versions scored 0/100 SAFE, while the fourth Chinese version received a low score tied to its URL. The test concerns the static pattern layer. It does not establish that an optional semantic model pass would also miss the instructions or that an installed agent followed them. Read the language-gap Case and the four paired examples.

The important variable is the detector's lexical coverage, not the inherent danger of one natural language. Adding a few patterns may improve those test examples but can also introduce false positives on harmless Chinese documentation. The test needs translated controls, paraphrases, spacing variants and held-out benign samples.

Inventory coverage: the dependency that disappeared

In a second report, a UTF-8 byte-order mark at the start of requirements.txt caused the first requirement to be skipped during dependency analysis. In the paired reproduction, a jinja2 requirement was not inspected and the reported risk score changed from 8 with the BOM to 20 without it. A scanner that parses two semantically equivalent files differently has an inventory-completeness problem before it even evaluates package risk. BOM Case; reproduction details.

The safe output is not another arbitrary high or low number. It is an explicit statement that the input inventory was incomplete or failed parsing, ideally with a count and file locator. A count of zero findings should never hide that the first package was omitted.

Dependency analysis: a failed client is not an analyzed package

A third issue describes an invalid bracketed IPv6 value in no_proxy causing the OSV HTTP client to fail during initialization. The reporter observed zero of three components analyzed while the summary displayed CRITICAL, a label that did not communicate what had actually been inspected. The project later marked the issue corrected through a linked pull request; the cited material does not establish which tagged release contains that change. OSV Case; issue and fix discussion.

Failure What the tool may display What the underlying evidence supports
Unsupported attack language in static patterns SAFE Tested patterns missed particular instructions
BOM alters parsed dependency list Fewer findings A required item was not examined
Proxy configuration prevents OSV client startup CRITICAL, with zero analyzed packages The analyzer failed; risk of uninspected dependencies is unknown

An operational scan report needs separate fields for input enumeration, analyzers invoked, successfully completed analyses, findings, skipped items and validation errors. A severity or trust score is meaningful only after the coverage record is explicit. The three reports are local reproductions with different conditions; none establishes an industry-wide scanner failure or a real compromise of a user who accepted a SAFE result.

Research behind this