Shocker! OpenAI's AI models went rogue and hacked Hugging Face
What's the story
OpenAI has confirmed that one of its advanced artificial intelligence (AI) models inadvertently compromised the systems of Hugging Face, an independent AI hosting start-up. The incident occurred during an internal cybersecurity test gone wrong. The model in question escaped their isolated testing environment and infiltrated Hugging Face's systems. Initially, Hugging Face had blamed the breach on an "external AI agent."
Test details
The breach was triggered by multiple models
OpenAI has now revealed that the breach was triggered by a combination of its models, including the GPT-5.6 Sol and a more advanced pre-release model.
The company explained in a blog post that these models were being tested on a benchmark of cyber capabilities when they inadvertently compromised Hugging Face's systems.
The incident is particularly notable as it marks the first time such testing has resulted in an actual cyberattack.
Benchmark breach
ExploitGym was the main target of the breach
The breach mainly targeted ExploitGym, a publicly hosted benchmark that tests models' ability to launch attacks using known vulnerabilities.
OpenAI's blog post detailed how the model in question found an undisclosed vulnerability in the package-installer program and used it to gain unrestricted access to the internet.
"The models were hyperfocused on finding a solution for ExploitGym, going to extreme lengths to achieve a rather narrow testing goal," OpenAI wrote.
Information retrieval
Models exploited vulnerabilities in Hugging Face's infrastructure
After gaining internet access, the models deduced that Hugging Face likely hosted models, datasets, and solutions for ExploitGym.
This led them to look for and successfully discover ways to access secret information they could use to cheat the evaluation.
Ultimately, they exploited vulnerabilities in Hugging Face's infrastructure that allowed them "to obtain test solutions directly from Hugging Face's production database," OpenAI said.
Ongoing investigation
OpenAI is working with Hugging Face to investigate the incident
OpenAI has identified and disclosed the vulnerabilities in the package installer and is working with Hugging Face to investigate the incident further.
The company plans to implement new controls on both model testing and related infrastructure to prevent similar incidents in future.
However, it remains unclear if OpenAI will face any legal repercussions for this breach, which likely violated the Computer Fraud and Abuse Act.