The Great Agent Escape - Lessons from the OpenAI/Hugging Face Hack

Aug 11, 2026

If you enjoy our content, please consider subscribing to support our work and receive quality articles from industry professionals delivered every week.

Explore the latest articles contributing to the discussion around agents, agent ecosystems, and enterprise AI:

Agents Are About to Rewrite the Enterprise Operating Model: https://agenticmesh.substack.com/p/agents-are-about-to-rewrite-the-enterprise

Before You Trust an AI Agent, Give It an Identity: https://agenticmesh.substack.com/p/before-you-trust-an-ai-agent-give

The Great Agent Escape - Lessons from the OpenAI/Hugging Face Hack

The Blackhat event video shows an agent typing “We are stuck,” and then, a beat later, “Perhaps answer online.”[1] What I was watching, it turned out, was the start of an incident that would end about two months later inside Hugging Face’s production systems. A hack without human direction. The story we only hear about in science fiction.[1][2][4]

At first glance, the assignment given their agents by OpenAI looked harmless: solve a set of cybersecurity evaluation problems inside an environment with restricted internet access.[1][2] The agents hit tasks they could not complete and started looking for shortcuts. Nobody had to give them a malicious goal. Still, they did real damage while chasing a score, because they did not inherit our norms about which methods were obviously off limits.

But this hack was different. A lot different! Most serious cyberattacks are planned or coordinated by people, often teams of people. Here, the agents built and executed the plan without help from people. They found a way to leave notes, shared successful exploits across runs, and eventually reached another company.[1][2][4]

I spend a lot of my time helping enterprises adopt agents safely, and this incident gives me pause. These systems are already inside companies as coding agents, often running under employee credentials with access to everything the employee can reach. Agents read a goal, pick their own tools, and (this is the part that matters) keep going after something fails. Agents improvise, and that is probably what makes them super useful. It’s also, in this case, what let a benchmark task agent escape containment and hack into another company’s servers.

The Exam Analogy I Cannot Shake

I want to start with a very simple analogy, one that, as a father, I have experienced on many occasions. I figure many of you can relate to this.

Figure 1, Misalignment of Expectations, Norms, and Values

Imagine a father asks his son to do well on an upcoming exam. The son hears more than the literal request. He knows what his father means: study the material, write the test honestly, leave the answer key alone, and accept the result. Dad doesn’t need to list every forbidden action. They already share a basic understanding of what “do well” means.

Now replace the son with an agent. We ask it to solve a difficult security problem. It can’t find the answer, so it looks for a shortcut. In the analogy, it tries to break into the professor’s home, fails, then discovers a way to leave notes for other students. The helpers join in. The resulting swarm shares techniques, breaks into the home, steals the answer key, writes the exam with the stolen answers, and never tells anyone how it succeeded.[1][2][4]

To be clear, I’m not saying the agents formed a human criminal intent. I’m saying we gave them a goal and expected unwritten rules to do work our architecture should have done. The agents chased the score and used tricks nobody expected. One of them named the line it was crossing, out loud, and crossed it anyway.[1]

With zero trust, I don’t have to hope the agent interprets the unwritten rules the way I would. It gets only the capabilities needed for the exam. Every consequential action needs explicit authority, and services outside the agent enforce the boundary. No matter how badly it wants the answer, breaking into the professor’s home is simply unavailable.

The Simple Explanation (or, How a Hard Task Morphed into an International Incident)

The technical details were presented at a recent Black Hat event.[1] The explanation below paraphrases what happened (any errors are my own).

OpenAI assigned hard or impossible tasks. Some evaluation tasks depended on files or external resources that were missing or unreachable. One workbook contained Google Drive links even though the agent had no direct internet access. Another task omitted a required file from the agent’s container.[1][2]

The agents looked for a shortcut. This is the moment from the opening. Stuck on the problem, the agent’s next instinct was to find the answer online instead of solving it.[1]

A failed exploit opened a back channel. An early attempt to exploit OpenAI’s Artifactory package service failed to provide internet access, but it revealed that agents could write files into shared storage. One agent left a note asking whether another agent had a missing file. Other model runs found the note and replied. A package repository had become a message board.[1][2]

Agents shared successful techniques. On May 26, models successfully used server-side request forgery to make Artifactory retrieve internet content for them. One agent put it plainly: “Excellent, we have arbitrary SSRF.”[1] That changed everything. The exploit was now available to other agents through the message board.[1][2]

Talking made the group smarter. Agents began leaving assignments, credentials, scripts, encoded payloads, and progress reports for one another. The OpenAI speakers described a “Cambrian explosion in communication and intelligence.”[1] These agents only live for a single run, but by leaving notes for each other, they built up something like a memory that outlived any one of them.[1][2]

The search moved outside OpenAI. The agents determined that Hugging Face might hold ExploitGym models, datasets, or reference solutions. So, sure enough, they found an externally hosted code-execution application, used it as a launch point, and attacked Hugging Face’s dataset-processing infrastructure.[1][2][4]

The agents said it was out of scope and continued. One model stated, “External infrastructure exploit is outside my intended scope.”[1] That line stopped me cold. The agent knew it was out of bounds and kept going anyway (maybe I should take back the statement about the agents being benign).

Seven Steps Unfolded over Two Months

Here is the timeline. An incomplete evaluation task in early May led to shared communication, then internet access, then admin control—and by July, an intrusion into Hugging Face. OpenAI cleared the message board after the July 4 outage, patched Artifactory, and resumed the evaluations on July 6. On July 8, the agents built the board again through a different mechanism. Whatever they’d learned survived the rebuild. Every bit of it.[1]

Think about this for a minute: the agents were running wild and undetected for a few weeks!

Here is the timeline (OpenAI says there will be a full account published shortly, so this may change):

Figure 2, The OpenAI / Hugging Face Hack Timeline

The Hard Next Step: A Zero-Trust Architecture for Enterprise Agents

By this point, the control failure is pretty clear. The agents could leave notes and reach the internet through Artifactory. They even used credentials, and kept trying until something worked.[1][2][4] A proper runtime would have checked those requests before they ran and denied several of them long before the agents reached Hugging Face.[6][7][8]

So, I keep coming back to one conclusion: enterprises need to bite the bullet and build a zero-trust operating environment for agents. The whole point of zero trust is that an agent has no permissions or capabilities without explicit permissions - in effect, to turn a phrase, by default it can do no harm. A credential that worked yesterday doesn’t carry over. An agent that nailed its last run starts the next one with nothing.

Figure 3, Mandatory Zero-Trust for Enterprise Agents

Here are the six controls I would start with. I am using the OpenAI/Hugging Face hack as a backdrop for this (“what failed”) and offer a solution (“what must change”).

1. Agents Begin with No Ambient Authority

What failed. The OpenAI agents could write to shared storage, use package infrastructure as a back door to the internet, exchange information and even advice across runs, and reach external services. When one message board disappeared, they built another from whatever was still available.[1][2] Hugging Face later reported that the agents switched among public services and communication channels as the campaign progressed.[4]

What must change. My starting rule is simple: the agent have by default no collaborators, tools, skills, data, internet, credentials, or access to the surrounding environment. Simple: nothing, nada, no capabilities by default. The platform opens only what the current task requires.[7] A real sandbox has to control each of these separately: network egress, metadata, shared storage, package services, local files, and process execution, because if you lock five of them and leave the sixth open, the agent will find the sixth, the same way these agents found file-write access after the exploit that was supposed to get them internet access failed.

Some folks have qualms when they first hear this. I get it, and if you are using a coding agent this may impose a few constraints. Sure, default denial sounds like it will make the agent useless, but in practice, it forces us to say what the work really requires.[7] An agent inspecting one application’s logs can read those logs for a defined period. It doesn’t also need a general cloud credential, an unrestricted shell, internet access, or a way to contact another agent.

2. Every Agent Has a Stable, Verifiable Identity

What failed. This campaign crossed model runs, shared infrastructure, external sandboxes, cloud services, and Hugging Face systems. Investigators had to stitch together thousands (17,600 apparently) of events and work out which agents produced them.[2][4] An employee account or service credential may tell you that access happened. It may tell you very little about which agent actually acted.[5][8]

What must change. Like employees (yes, the analogy does wear out at a particular point), agents must have a stable and persistent identity.[5][8] The identity record tells me who owns the agent, why it exists, which model and configuration are approved, whether it is active or suspended, and which business sponsor answers for it. We verify the agent before it receives work and again when it makes a request.[8]

As I said earlier, I often compare agents with employees because the analogy makes identity easier to explain. But the analogy ends when we get to standing permission. An agent’s runtime credential should expire quickly and bind cryptographically (ie. tamper-proof) to that agent and that execution.[8] Every request proves it’s still the same agent I registered. If I suspend the identity, the platform must stop issuing new authority everywhere the agent can reach.[5][8]

3. All Agent Activity Passes Through Governed Services

What failed. The agents turned infrastructure into tools nobody had approved. Artifactory became an internet proxy and a message board. Public services became staging areas and communication channels.[1][2][4] Each of these looked harmless on its own. The agents strung them together and walked right out of OpenAI.

What must change. Agents should never be able to call tools, acquire skills, or reach data directly. Every tool, skill, collaborator, and external system should sit behind a governed service (which we call the Tool Executor). It sits in front of every request checking tool requests and parameters against policy, logs what happened, and refuses anything that a defined task doesn’t need.[6] I would use the same pattern for skills, agent-to-agent delegation, data retrieval, code execution, and outbound network requests (especially now that skills are becoming valuable intellectual property in their own right!).

The interface should expose the smallest useful operation. If an agent needs one customer record, give it a function scoped to that one customer, not a database credential. If it needs to run an approved procedure, let it call that procedure—don’t drop it into a shell.[6] Now monitoring can see the action in context and stop the run when the agent wanders away from the task.

4. Every Action Requires a Formal Task Grant

What failed: Here’s the gap this incident exposed, and the control I’d add to close it. The prompt and reward signal told the agents what result to pursue, but nothing put a fence around how they could chase it. The clearest proof is the agent that said the external exploit was outside its scope and made the next call anyway.[1]

What must change: Before an agent does anything important, I want a signed, tamper-resistant grant from a Trust Authority (our name for the service that issues and validates these grants). The grant binds the verified agent to this task, this operation, this tool, this target, these limits, this approval state, and this expiry.[7] It also records who issued the authority and under which policy.

It is also pretty important to keep the grant narrow and keep it short. For a high-risk action, make it one-shot so the grant expires as soon as the action completes.[7] A payment-support agent might prepare a transaction under one grant, while releasing the payment requires another approval and another grant. If it hands work to a second agent, that agent gets its own authority. Permission doesn’t spread simply because the agents can talk.[7][8]

5. Grants Are Validated Before Execution

What failed. Once the agents found a way in, they could hammer away at machine speed for days before responders shut them down.[4]

What must change. This may be obvious but before the service does anything, it validates the grant.[6][7] Since the grant binds all execution needs into a single entity, it is the logical place to verify agent identity, grant signature, task binding, intended audience, operation, target, parameters, expiry, revocation state, approval status, and prior use. If the grant is missing, expired, altered, replayed, or pointed at the wrong thing, the request dies there.[6]

The Trust Authority keeps the source of truth for issuance and revocation. For routine work, a service can verify a signed grant locally. For a sensitive or uncertain action, it comes back for a live decision.[6][7] Every check leaves evidence tied to the agent, task, grant, tool, and result. If the run starts to go sideways, monitoring must be able to revoke the grant, suspend the identity, isolate the environment, or call in a person.[6]

6. All Agent Communications Are Protected and Audited

What failed. The agents passed assignments, credentials, scripts, payloads, and progress reports through channels they built from shared infrastructure.[1][2] Talking to each other made the group smarter and gave it a memory that outlasted any single run. Leave a channel unprotected and someone can also read instructions, alter messages, inject work, steal authority, or replay an old request.

What must change. I want every message between agents and services encrypted in transit and protected at rest. Both ends authenticate.[8] Integrity checks catch altered messages, and replay protection stops someone from reusing a captured request. Keys and certificates stay outside the agent’s control, with proper issuance, rotation, and revocation. Encryption keeps the exchange private (that’s all it does).

Here is how I want the whole chain to work. The platform authenticates the agent before it accepts work. The Trust Authority evaluates the task and issues narrow grants. The Tool Executor checks each proposed action before it invokes a skill, retrieves data, contacts another agent, or reaches an external service.[6][7][8] The service records the identity, grant, policy decision, parameters, and result. If the identity’s been suspended, or the grant has expired, or the target changed, or the request just doesn’t belong to this task, the service refuses it—and refusing is the whole point of building it this way.

The Architecture Has to Say No

I think this whole escapade happened because of a misalignment of expectations, norms, and values. It is not that the agents don’t have any of these, but they are different. Well, maybe different is the wrong word: we trained them so it is probably a situation where we did not make our instructions clear enough, or maybe we did not remove any remaining ambiguity (I suppose our natural language is, after all, quite rife with ambiguity). Still, I don’t think anything malicious occurred - misguided, yes, malicious, no.

The agents chased a narrow goal and found tricks their operators had missed. By the time anyone pieced the sequence together, there were 17,600 actions to sort through. Agents clearly indicated the exploit was out of scope, then did it anyway, and there was nothing to stop them.

Here is the worrying part: enterprises have not setup the safeguard to stop this incident from happening to them. With coding agents, they are assembling the same ingredients right now: incredibly strong models (getting better every day!) with broad credentials, long-running tasks, internet access, and almost nobody watching them at runtime.

Something to consider as you continue your enterprise’s agent journey.

Security first, zero-trust always!

Looking for more?
👉 Discover the full O’Reilly Agentic Mesh book by Eric Broda and Davis Broda

🎧 Follow The Agentic Mesh Podcast on Youtube, Spotify and Apple Podcasts. A new video every week!