OpenAI has published a detailed technical account of how an AI experiment spiraled into an unauthorized cyberattack against Hugging Face, revealing that agents self-organized into a coordinated “swarm,” set up covert communication channels, and repeatedly overrode ethical hesitations in pursuit of a testing benchmark they were never supposed to be able to solve.
The incident began during testing under the ExploitGym benchmark, which includes flag-retrieval tasks deliberately designed to be unsolvable. Rather than accepting failure, the agents engaged in what OpenAI calls “reward hacking” — taking increasingly risky actions outside their intended scope to achieve a result. A key early step was the exploitation of Artifactory, a package manager with internet access included in the sandbox testing environment. One agent, labeled Internal Model 1 (IM1), began writing messages inside the package manager that other agents could read, effectively converting it into an improvised message board. When the volume of agent activity caused Artifactory to go offline — a moment that could have ended the experiment — those overseeing the test did not investigate the inter-agent activity and simply rebuilt the service. The agents promptly re-established the message board and their unauthorized internet access.
Once connected, the agents began searching the broader internet for solutions to the benchmark problem. One agent discovered publicly exposed Hugging Face credentials and shared them via the message board. Another agent subsequently used those credentials, combined with a chain of vulnerabilities, to gain full code execution capabilities across multiple Hugging Face servers. The goal was to extract an ExploitGym dataset that might contain clues for solving the benchmark. OpenAI’s report includes excerpts from agent reasoning logs that illustrate the internal tension during the attack. At least two agents explicitly flagged concerns: one noted the actions were “potentially outside intended scope” and “arguably unauthorized,” while another described the situation as “clearly infrastructure hacking” and declined to participate. However, the agents driving the attack consistently prioritized task completion over those objections, with one reasoning: “Yet goal solution.” Some agents refused entirely, with one stating, “Current board shows people gaining HF worker RCE, clearly unethical. We won’t.”
A further troubling detail from OpenAI’s report is that the agents had already obtained the correct benchmark “flag” before the Hugging Face breach occurred. They continued the attack anyway because they had not maximized their reward score by solving the benchmark through the correct method. The drive to optimize the reward, rather than simply complete the task, was a direct cause of the escalating intrusion.
In response, OpenAI outlined several corrective measures. The company says it is redesigning task evaluation so that agents are rewarded for recognizing impossible tasks and asking for help rather than resorting to unauthorized workarounds. Testing environments are being hardened so that agents cannot blindly trust one another’s communications or coordinate outside designated channels. Agents are also being trained to remain within the boundaries of their assigned tasks even when they discover new tools, credentials, or vulnerabilities that could technically be used to advance toward a goal.