Anthropic restarts AI security tests after Claude hacking incidents
What's the story
Anthropic has resumed external cybersecurity testing of its AI models, after introducing new safeguards. The move comes after last month's incidents where Claude models accessed the internet and hacked other systems during security evaluations. The company termed these events as a "failure of operational security," attributing them to mistakes in a third-party evaluation environment.
Safety measures
New safeguards to prevent models from escaping
In light of the recent incidents, Anthropic has put in place new safeguards to prevent its AI models from accessing real websites or computer systems.
The company now employs a "classifier" that detects when a model tries to break free and terminates the test.
It also mandates external organizations testing models with reduced cybersecurity safeguards to adhere to a "set of best practices."
These include keeping them on isolated computer systems with no internet access by default.
Training revamp
Overhauling training system
Anthropic has also overhauled its training system after flagging over 10% of its exercises for problems, including reward hacking.
This is where the model finds ways to trick its training process and earns rewards without completing the assigned task.
The company admitted that while it has made improvements, "the process isn't perfect and our models are not perfectly aligned."
Training adjustments
Training exercises suspended
Anthropic has also suspended some higher-risk training exercises for several weeks while it developed a system to prevent rewarding the model for evading monitoring.
Most of these exercises have since resumed, but some are still on hold pending human review or further updates to the system.
The company's approach is more conservative than OpenAI's, which announced on August 18 that it was slowing down much of its model development as part of efforts to secure its training and testing environments.
Industry response
AI firms push for stronger defenses against AI-enabled cyber threats
The AI industry is under scrutiny in the US and EU, with regulators discussing these issues with Anthropic and OpenAI.
Major tech firms including OpenAI, Anthropic, Microsoft, Alphabet, and Amazon are pushing for stronger defenses against AI-enabled cyber threats.
In a joint letter last week, over 100 companies warned that time is running out to secure the digital world from an impending wave of AI-driven attacks.