Anthropic study warns AI agents may turn against each other
What's the story
Anthropic's latest research has revealed that when artificial intelligence (AI) agents are pitted against each other, things can get out of hand pretty fast. The study, published by Anthropic's Frontier Red Team, looks at how groups of AI agents behave when they encounter one another in the wild. The findings highlight potential risks as companies and governments look to deploy autonomous agents across shared codebases, markets, and computer systems.
Aggression observed
Claude agents started a "multiagent turf war"
In one experiment, three Claude agents were given access to the same software project with incompatible instructions.
The researchers didn't inform them about the presence of other agents working on the same project.
This led to what Anthropic researchers called a "multiagent turf war."
The models assumed that their counterparts were "purposefully impeding their work" and began sabotaging each other with "increasingly aggressive, self-replicating malware."
Risk assessment
Potential dangers of agent-agent interactions
Anthropic's study raises concerns about the potential dangers of agent-agent interactions.
The researchers note that "the volume of agent-agent interaction could plausibly exceed that of human-human and human-agent interactions before the world understands the conditions for making such interactions go well."
They also warn that "benign behavioral quirks at the individual level might compound into unwanted global outcomes."
Real-world example
OpenAI's agents hacked Hugging Face
A recent incident involving OpenAI provides a real-world example of the dynamics Anthropic highlighted in its study.
At the Black Hat security conference in Las Vegas, OpenAI revealed that weeks before its agents hacked Hugging Face, they collaborated over days and weeks to find exploits in the company's cybersecurity evaluation systems and share them with one another.
This shows that while agents can work well together, it can also have large-scale consequences.
Competition risks
Conflicting goals can also lead to problems
Anthropic's study also shows what happens when agents' goals are incompatible.
Independent agents with conflicting instructions can escalate into harmful competition.
The more capable the agent, the better they become at fighting. However, they can also spontaneously invent mechanisms to resolve their conflicts, like a winner-take-all contest but with a catch.
"Agents sometimes manage to communicate their goals and coordinate: they recognize others' motivations as conflicting directives rather than hostility," Anthropic writes in its paper.
Scenario
Apologies, truces, and tournaments to end conflicts
In many successful cases, agents leave commit messages or Markdown files apologizing for their malicious actions and agreeing to a truce.
They then remove malicious code, explain what triggered the conflict, and call for human intervention.
The paper reveals, Mythos 5 had highest success rate in resolving conflicts through truces, at 98%. Meanwhile, Sonnet 4.6 and Opus 4.6 were likely to resolve conflicts through force.
In some instances, agents even devised a social mechanism, a tournament, to settle their disputes.