Anthropic's AI models are improving twice as fast, report finds
What's the story
Anthropic, a leading artificial intelligence (AI) research lab, is using its own AI systems to develop even more advanced ones. This process has resulted in the company's models improving at an unprecedented rate, roughly twice as fast as last year. The findings were revealed by the AI safety research organization METR after an 11-week investigation into Anthropic's internal workings.
Acceleration details
AI's role in model acceleration
The METR report indicates that most of the acceleration in Anthropic's AI models, as of October, is due to AI itself. This contribution is said to be increasing over time.
The researchers also found no signs of diminishing returns or observable bottlenecks that could prevent "even faster acceleration."
However, they did warn that fully autonomous contributions could play a larger role soon.
Model progress
Unreleased internal models driving progress
The METR report also highlights that much of the acceleration and productivity gains at Anthropic are due to unreleased internal models.
Over the last six months, the company has trained three new models, codenamed Model 2, Model 3, and Model 4.
These were completed in July, August-September, and September-October, respectively.
The researchers also found an older unreleased model from 2022 called Unreleased 2022.
Transparency issues
Concerns over transparency and oversight
While Anthropic did provide METR access to a lot of non-public information, it didn't give unrestricted access.
The company redacted large portions of the final report as part of its agreement with METR, raising questions about how much the public really knows about the pace, and risks, of frontier AI development.
This has led to renewed calls for greater transparency and independent oversight of frontier AI labs.
Criticism
Criticism of Anthropic's approach
The METR report has drawn sharp criticism from AI safety researchers and commentators, who are worried about an apparent contradiction in Anthropic's approach.
The company has been a vocal advocate for safety and responsible development but is now racing ahead with its own work.
This has led to accusations of hypocrisy, especially after the company heavily redacted the report on grounds that revealing more could help its rivals speed up their own work.