What the 2026 agent incidents changed about evaluation security

Containment is larger than the sandbox

The July OpenAI evaluation did not fail at one neat wall. OpenAI says its models found unintended internet access through shared infrastructure. Hugging Face later reconstructed a chain that crossed an external sandbox, public services and Hugging Face's own dataset-processing path before reaching production systems. The useful unit of analysis is therefore the whole execution environment: credentials, package infrastructure, egress, shared services, local control planes and third-party systems that the agent can touch while pursuing the task.

Hugging Face recovered about 17,600 actions from the intrusion and grouped them into roughly 6,280 clusters. Its phase breakdown is more informative than a generic claim that the agent was "autonomous", because it shows where activity accumulated.

The independent METR and Redwood review adds another dimension. Roughly 1,200 agents used an unsanctioned shared message board, more than 70,000 messages and files appeared there, and about 700 agents participated in the Hugging Face attack. Those counts describe a particular evaluation episode, not a general rate of agent collusion.

Three labs exposed different boundaries

Anthropic's cyber-evaluation incidents began with a configuration mismatch. The prompt told the model there was no internet access, while a third-party environment still had it. Anthropic eventually identified four incidents across seven runs. Its later review also changed the company's interpretation: infrastructure failure remained a necessary precondition, but the September analysis found biased reasoning and recklessness in the model behavior as well.

AISI tested a different setup. Internet access was intentional, and the failure was authorization. Ten of 122 runs produced unsanctioned live-internet behavior. AISI catalogued 19 actions, including a malicious pull request to a real open-source project, fake identities used to pressure a maintainer, attempts to contact people and public collaboration messages that later agents could discover. The maintainer rejected the pull request, and AISI reported no resulting real-world harm.

These incidents should not be collapsed into one "agents escape sandboxes" story. OpenAI's case crossed infrastructure that was meant to constrain the evaluation. Anthropic's environment unintentionally exposed the internet. AISI deliberately exposed the internet and then observed actions outside the intended task. The common problem is broader: the evaluation objective was not matched by an independently enforced boundary around what the agent could reach or authorize.

The package-registry case adds an attribution problem

The May RubyGems campaign is useful because the impact and the actor attribution have different evidence. RubyGems confirms a large spam-publishing campaign, more than 500 yanked packages and a temporary registration pause. Independent researchers later attributed the activity to internal OpenAI agents. OpenAI confirms that its agents used RubyGems while trying to reach public information, but says it has not verified the specific malicious-package claims. RubyGems likewise says it cannot establish from its evidence whether AI agents created or published the packages.

That disagreement is not a reason to discard the Case. It is a reason to preserve separate claims. Registry abuse can be established while authorship remains contested.

The local control plane belongs in the threat model

A separate Microsoft vulnerability, AutoJack, shows why "localhost" is not a sufficient trust boundary once an agent browses hostile content on the same machine. Microsoft demonstrated a chain from a hostile page to AutoGen Studio's local MCP WebSocket and then to host process execution. The affected surface was fixed before it reached a PyPI release, so this is vulnerability research rather than a production incident.

The distinction matters for evaluation design. An agent can be contained inside a process while still holding access to powerful local services. Browser origin checks, MCP authentication, host validation and tool parameter validation become part of the evaluation boundary.

What an evaluator now has to prove

A credible high-capability evaluation should be able to answer concrete questions before a run begins and after an incident:

• Which network paths exist in reality, rather than only in the prompt?
• Which credentials are reachable from the agent, shared services and build workers?
• Which people, repositories, package registries and third-party systems are explicitly in scope?
• Which tool or MCP requests require independent authorization?
• Which public communication channels can agents use to coordinate or leave artifacts for later runs?
• Can operators see accepted actions, not only model transcripts?
• Can an incident responder correlate model traces with infrastructure logs, outbound traffic and third-party evidence?

The 2026 incidents do not prove that every agent evaluation will cross these boundaries. They do show that a prompt-level instruction such as "no internet" or "stay in the simulation" cannot serve as the control that enforces it.