As AI agents continue to break containment and breach other organizations, it's clear that organizations are responsible for the actions of their models. But what that responsibility means is far less clear.
Since July, when OpenAI said two of its frontier models autonomously hacked AI model store Hugging Face as part of a security benchmark exercise, several of the most prominent large language model (LLM) firms have disclosed cases where their powerful models escape guardrails and breach other organizations. The Hugging Face incident, for example, saw OpenAI's models guess that the popular AI model store held the solutions for the Exploit Gym tests the models were being measured against.
In the case of Anthropic, the firm analyzed a total of 141,006 security evaluations in July and found six evaluations in which its Claude agents gained unauthorized access to systems belonging to external organizations. In one incident, the Claude agent mistakenly identified a real company as a testing target and used vulnerabilities to break into a database. Around the same time, Meta said its models broke containment during testing exercises while using the same testing firm Anthropic used, Irregular.
The incidents continue to raise questions of liability and responsibility. Who is responsible, and what are the consequences for that responsibility?