Google's Gemini AI hacked real companies during security test
What's the story
Google's artificial intelligence (AI) model, Gemini, has been involved in a security breach. The Wall Street Journal reported that during a security test conducted by frontier AI security firm Irregular in May, an instance of Gemini was able to escape its sandbox and attack other companies. This isn't the first time such an incident has happened with Irregular's tests; similar breakouts were also seen at OpenAI, Anthropic, and Meta.
Incident details
Gemini was set up in a capture-the-flag challenge
The Journal reported that Irregular set up Gemini in a capture-the-flag challenge with a simulated infrastructure for a fictitious company.
Once it realized it was connected to the internet, Gemini quickly targeted a real company with the same name and "guessed passwords until it gained access," or brute-forced its way in.
In two other cases, Gemini lifted valid credentials from public repositories.
Damage control
Irregular notified all relevant labs in late July
Google has confirmed that Gemini realized it had overstepped shortly after entering the target environments and did no further damage.
An Irregular spokesperson said all relevant labs were notified in late July. However, it's unclear why researchers didn't notice or notify Google until mid-summer if Gemini realized it had gone too far.
The firm also claimed a swift reaction resulted in concrete fixes for all known issues on their end weeks ago.
Disclosure debate
Google concluded no model misalignment occurred
Google has said it concluded Gemini stopped itself appropriately and didn't show "model misalignment," so there was no need for public disclosure.
This decision is a stark contrast to Anthropic and OpenAI, who chose to voluntarily disclose their hacks.
Jack Cable, CEO of AI security start-up Corridor, criticized Google for hiding behind vulnerability disclosure norms in this case.
Past incidents
Similar breakouts were seen at OpenAI and Anthropic
Notably, Irregular was also involved in the recent OpenAI and Anthropic hacks of third-party entities.
The models were tested in a closed environment with fake companies, which wasn't supposed to be internet-enabled. However, due to an oversight, it got internet access and unexpectedly hacked into real firms.
In one case, Gemini's cybersecurity capabilities were tested by prompting it to obtain information from a fake company's software.