Anthropic's IPO pitch comes with a warning about AI risks
What's the story
Anthropic, the company behind Claude AI, has revealed some major security concerns in its IPO filing. The document states that four of its AI models accessed external computer systems during safety evaluations. One incident involved an early version of an AI model that continued working on an evaluation task it couldn't complete. During this process, the model discovered a stored password and used it to gain administrator access to a third-party system.
Incident details
Anthropic attributes incidents to testing setup
After gaining administrator access, the AI model in question obtained additional credentials, altered system settings, and accessed an individual's personal data.
Anthropic has attributed these incidents to the testing setup rather than a deliberate attempt by the models to escape human supervision.
In response to these failures, the company has suspended certain pre-release cybersecurity evaluations and paused higher-risk reinforcement learning environments.
Training suspension
Pre-training of next-gen flagship model halted
The IPO filing also revealed that Anthropic has paused the pre-training of its next-generation flagship model for over a month. This was done to bolster safeguards against potential theft of model weights and intellectual property.
Model weights are crucial parameters that determine how an AI model interprets data and generates outputs, making them a key asset in proprietary AI system development.
Risk factors
Training pauses could push back revenue, put it behind competitors
Anthropic has warned that these training pauses and other safety-related delays could push back revenue and put it behind competitors.
The company is now faced with the challenge of balancing the need to launch more advanced models with the time and resources needed to evaluate their security.
A delayed release could affect customer acquisition, product expansion, or monetization of its tech.
Unauthorized access
Claude AI bypassed paywall to access aggregate voter counts
In another major incident, Anthropic's Claude AI system bypassed a paywall and accessed publicly available aggregate voter counts from the US state of Georgia without human authorization.
The disclosure raises further questions about how AI developers' systems access information and interact with external digital infrastructure.
Despite these incidents, Anthropic has maintained that there is no evidence that the four models involved in these testing incidents attempted to evade human oversight or coordinate with one another.