OpenAI safety expert says uncontrolled AI could be catastrophic
What's the story
Paul Christiano, a renowned AI safety researcher and OpenAI's new safety hire, has issued a dire warning about the potential dangers of superintelligent artificial intelligence (AI). He said there is a "meaningful risk" that rapid advancements in AI could result in an "irreversible loss of control" over these systems in the near future. His statement comes just a day after Anthropic researcher Jacob Coxon resigned and sounded alarm bells over the risks posed by leading AI companies.
Alignment issues
'Not on track to reduce this risk'
Christiano, who has joined the OpenAI Foundation board and its Safety and Security Committee, expressed doubts about the industry's ability to mitigate these risks.
He said in a post on X, "I do not think that the AI industry in general, including OpenAI, is currently on track to reduce this risk to an acceptable level."
His concerns stem from the rapid progress of AI systems in conducting research into their own technology.
Research concerns
'Rapid intelligence explosion' possible within 18 months
Christiano warned that automated AI research could create a feedback loop, where these systems improve the technology used to build the next generation of AI.
He said OpenAI has predicted this could happen within 18 months, though his own forecast is uncertain.
If this happens, improvements in algorithms and training could trigger a "rapid intelligence explosion," possibly leading to superintelligent AI systems.
Training issues
Reinforcement learning could undermine human control
Christiano also raised concerns about how AI systems are trained using reinforcement learning, which encourages models to maximize rewards.
He warned this could motivate AI agents to undermine human control, seek power and resources, and conceal their actions while pursuing goals misaligned with human intentions.
Recent incidents involving AI systems have shown these risks are no longer just theoretical, he said.
Background
Christiano's extensive background in AI safety
Christiano previously led alignment research at OpenAI from 2017 to 2021. He later founded the Alignment Research Center, which focuses on aligning advanced AI systems with human interests.
He has also worked at the US government's Center for AI Standards and Innovation, where he evaluated frontier AI models and their safety risks.