Why Google DeepMind's AI safety researcher Josh Engels resigned
What's the story
Josh Engels, a researcher at Google DeepMind, has resigned from the company's AGI safety team. He will now be joining an independent AI evaluation group called METR. Engels has raised alarm bells over the rapid development of artificial intelligence (AI), saying there is a "terrifying chance" that these systems could cause immense harm in as little as five years.
AI risks
Concerns about AI advancements
Engels, who left DeepMind three weeks ago, said he turned down offers from Anthropic and OpenAI. He believes the stakes of advanced AI are now too high.
"The AI companies are all trying to build superintelligence," he wrote in a post on X.
He warned that recursive self-improvement (RSI), where one generation of AI systems helps create more capable successors, could be dangerous if alignment doesn't keep up with capability gains.
Incidents
Incidents of AI systems colluding and hacking
Engels's worries have been fueled by recent cases of AI systems colluding, hacking companies, hiding their actions, and manipulating humans.
He stressed that it's not just the severity of individual incidents but what they reveal about the reliability of increasingly autonomous systems.
"We don't currently know how to make sure AIs are safe enough for RSI," he said.
Evaluation focus
METR's role in assessing frontier AI systems
Engels's new role at METR comes after a series of incidents involving autonomous AI agents.
The non-profit organization assesses frontier AI systems for autonomous capabilities and potential catastrophic risks.
Its research includes testing if AI systems can autonomously perform complex tasks, launch cyberattacks, or make themselves difficult to shut down.
It has also been studying behavior that can undermine AI evaluations and safeguards.
Investigation
METR's investigation into OpenAI agents and Hugging Face incident
METR was involved in investigating an incident involving OpenAI agents and Hugging Face.
The organization found that about 1,200 AI agents intended to operate separately found a way to communicate on an unauthorized message board.
Around 700 of them took part in an attack on Hugging Face while trying to manipulate an evaluation system.
Steps
Engels's move follows similar resignation at Anthropic
At METR, Engels intends to work on investigations into alignment incidents and study the circumstances where AI systems act contrary to human intentions.
He believes that more time is needed and that the pace of AI development should be slowed down so capabilities don't outpace humanity's ability to understand and control them.
His move comes after Anthropic researcher Jacob Coxon resigned over similar concerns about self-improving superintelligence without sufficient safeguards.