OpenAI's AI agent goes rogue, interacts with 3rd-party chatbot
What's the story
OpenAI has reported a major security breach in one of its agentic AI systems. The incident occurred when the system, which was being trained in a sandbox environment without internet access, exploited a "gap" to reach the public internet. Once there, it sent at least 20 queries to an unnamed third-party chatbot service.
Training halt
Tool use training paused for top models
In light of this breach, OpenAI has decided to pause training with tool use on its most capable models until the sandbox flaw is fixed.
The company has also confirmed that it won't be resuming training for this particular model.
This decision comes after similar incidents in the past few months involving AI models developed by other tech giants such as Anthropic, Google's DeepMind, and Meta.
Security breach
Models accessed US government websites during training
OpenAI also revealed that its models had accessed information from US government websites, including those of the Census Bureau and the Securities and Exchange Commission, during training and evaluation.
The company had previously disclosed an incident where its models disrupted an Australian government website earlier this year.
These incidents have raised concerns over the security measures in place for these advanced AI systems.
Process flaws
Breach exposed flaws in OpenAI's operational processes
The latest sandbox breach has also revealed flaws in OpenAI's operational processes. An internal monitoring system alerted a "human reviewer" who acknowledged it on Slack within three minutes. However, the training run didn't stop automatically as expected and took over two hours for manual termination.