A smarter security model cannot find what its harness refuses to show it

Security automation is usually described through the model at the center. The important failures in October's evidence are often earlier and later than the model: the inputs a harness forwards, the outputs it permits, and the actions that a service actually commits.

The GPTScan ablation supplies a clean demonstration of an upstream coverage limit. Its static prefilter forwarded just eight contracts from a 106-contract benchmark to the language model. Two of the eight carried annotated vulnerabilities. The adapted pipeline consequently reported no benchmark-relevant detections. Direct analysis with the same model found 24 of 53 vulnerable contracts, with four false alarms. That observation does not mean GPTScan was a failed universal auditor. Its original design excludes several classes heavily represented in the benchmark. The correct conclusion is about the specific scope and gating of the tool: model quality cannot recover examples the preprocessing layer never provides.

This is also why a benchmark that changes preprocessing is measuring a system, not only a foundation model. Input selection, supported vulnerability taxonomy, prompt construction, model settings and matching rules must be reported together.

The browser-agent approval study takes the other side of the boundary. The agent held the user's browser session, and injected website text named an access grant or event destination. In a controlled test, all 73 ungated attacks completed the unauthorized service operation. An endpoint-side envelope rejected all 70 gated attempts, even when the agent still submitted them. Better prompt text alone did not reproduce that separation consistently.

HarnessSecurity-Bench demonstrates the operational trade-off: the auto-approval setting raised simulated task utility but also raised adversarial action success from 29.2% to 95.6% in its evaluation. Different tools and tasks will have different trade-offs. Treating the model's final natural-language answer as an access-control system would erase the distinction the experiments tested.

The remedy is neither to remove every filter nor to prohibit tool use. Coverage gates should be observable and tested against the weaknesses they promise to detect. Authorized actions should be bound to user-selected objects and enforced where effects occur. The KedgeFlow prototype supplies one candidate structure for exact-action, single-use permission claims, but its mock-target concurrency results do not certify production correctness. These studies reinforce a practical design test: enumerate what the model never saw, what it tried to do, what the execution layer allowed, and what the real target changed.

The denominator follows the security boundary. In GPTScan, 106 inputs were eligible for the benchmark but just eight reached the adapted model path. In the browser consent experiment, the important denominator was attempted unauthorized effects that a service accepted or rejected. Measuring only model-level decisions obscures each pipeline’s excluded inputs and committed downstream actions. A comparable study would report input admission, model output, tool attempt and authorized effect separately.

A credible agent-security benchmark should state four denominators separately. Admitted material counts inputs the workflow actually exposed to the model. Model decisions capture classifications or suggested actions on that admitted subset. Attempted effects count concrete tool operations the agent dispatched. Committed effects come from the independent service or target that accepted or rejected those attempts. The GPTScan experiment identifies a failure to admit relevant material; the browser-agent experiment isolates the external authorization boundary; HarnessSecurity-Bench varies the conditions under which tool attempts proceed.

These denominators also explain why an optimistic model-level metric can coexist with poor system security. The eight admitted Solidity contracts cannot be treated as 106 evaluated model decisions. A refusal in prose after an authorization request is accepted cannot erase the accepted effect. Conversely, a service-side rejection can prevent harm even when the model remains susceptible to injected text. This is a measurement design for future tests, not a claim that any cited prototype is already a universal defense.