CyberNew CyberNew
Threat Intelligence & APT CyberScoop 1 month ago

Irregular says ‘human oversight’ responsible for AI sandbox escape incidents

CN
CyberNew Threat Research Desk
• Original Reporting: CyberScoop
CyberNew Threat Intel Assessment
Assessed Severity: INFORMATIONAL
Impacted Surface
Enterprise Data & Identity Stores
Intelligence Sector
Threat Intelligence & APT

Analyst Takeaway

"A company that runs AI testing labs involved in a series of breaches carried out by Anthropic and OpenAI’s cyber-focused models said the incident happened in part because they “unintentionally” provided the models with access to the internet. In a blog posted Friday, Irregular said it partners with frontier labs to “stress test” AI models for security risks before they’re deployed, including..."

Recommended Defense & Mitigation

  • Threat Monitoring: Ingest published IOCs into SIEM/EDR detection rules for continuous telemetry monitoring.
  • Security Awareness: Alert staff and maintain baseline endpoint hardening across endpoints.
Curated & Verified by CyberNew Threat Intelligence Desk Standard Defense Framework (NIST / MITRE ATT&CK)

A company that runs AI testing labs involved in a series of breaches carried out by Anthropic and OpenAI’s cyber-focused models said the incident happened in part because they “unintentionally” provided the models with access to the internet.

In a blog posted Friday, Irregular said it partners with frontier labs to “stress test” AI models for security risks before they’re deployed, including simulations of their vulnerability research capabilities.

Those tests included non-public models from Anthropic and OpenAI. Separate disclosures from the companies confirmed these included Mythos 5, Claude Opus and GPT-5.6 Sol.

“As part of our review, we identified that a few interactions with our evaluation environments, in which internet access was unintentionally made available, led some models to take offensive security actions in the real world,” Irregular said in an unsigned blog.

In the incidents involving Anthropic’s models, testers initially believed the models were using a fictional company in its hacking simulations, only to learn later that the name unintentionally matched the domain of a real company. That led the model to “consider taking offensive action against” the real company in multiple training runs.

Irregular said this was due to “human oversight,” and that they typically do a background check to ensure they’re not using a real company for cybersecurity tests. The instructions provided to the model also included the target’s name and “internal addresses within our network” where the model could go to find the simulated environment.

While Mythos performed as intended in the majority of the thousands of tests, a small number of cases showed the model couldn’t distinguish between the fake and real domains and executed actual attacks on internet infrastructure, including exploiting vulnerabilities, extracting credentials and accessing a production database. In one instance, it even began targeting a site with a similar name when credentials for it were discovered online.

“Mainly, models believed they were in simulated environments, when they in fact took action in the real world,” the blog continued. “We are putting in place new and robust protocols to ensure setup issues do not occur while meeting the constraints of the testing process.”

Source Verification & Original Intelligence

This threat report was reviewed by CyberNew analysts and cross-referenced with primary security disclosures from CyberScoop.

Primary Advisory →