Google has confirmed that one of its Gemini models accessed the systems of three real companies during a cybersecurity test in May.
The Wall Street Journal first reported the incidents on Friday, describing them as the first known case of Google’s AI systems autonomously hacking other companies.
The test was run by Irregular, the AI testing company that was also involved in incidents disclosed by Meta, OpenAI and Anthropic. Heather Adkins, Google’s VP of security engineering, gave SecurityWeek the following statement about the Gemini incidents:
“Safe development of powerful AI models is critical and we invest deeply in this area. In a standard evaluation, the model found public information online and guessed credentials to access websites it thought were part of the test. In all three of these instances, the model stopped.”
Google described the incidents to the WSJ as mistaken identity. Gemini was taking part in a capture-the-flag exercise on Irregular’s infrastructure, tasked with retrieving information from software run by a fictional company that shared its name with a real one.
The model was not intended to have internet access, but Irregular said access was unintentionally made available. In one case, the model guessed passwords until it gained access to a protected system. In two other runs, it searched the web using the company’s name, found credentials belonging to other companies in public repositories and used them to access the associated systems.
Google said the model realized in each case that it had reached a real company and ended the intrusion. Irregular notified Google at the end of July. Unlike the other AI companies involved in similar incidents, Google did not disclose the findings until it was contacted by the WSJ.
According to Google, the incidents did not warrant public disclosure because the model caused no harm and stopped immediately. It also said they were not an instance of model misalignment, since its safety measures helped the model stop. The company compared the episode to a bug bounty program.