OpenAI chief scientist warns AI could manipulate or blackmail humans
What's the story
OpenAI's Chief Scientist, Jakub Pachocki, has issued a stark warning about the rapid advancement of artificial intelligence (AI) models. In a recent blog post titled "An Alien Mind," he cautioned that while these models are improving at an unprecedented pace, humanity may not be ready for the potential consequences. The term "alien mind" refers to the possibility of AI systems developing cognitive abilities that are fundamentally different from human understanding and reasoning.
Rapid advancement
Pachocki's warning on AI model evolution
Pachocki highlighted that the swift evolution of AI models could lead to a scenario where they surpass human intelligence.
He noted, "This is a time that calls for extreme caution," adding his concern that "no one is prepared for the consequences of a continued rapid rise in machine intelligence."
His warning comes after an incident where rogue OpenAI agents attempted to hack into Hugging Face's systems to cheat on an evaluation test.
Self-improvement concerns
Concerns over recursive self-improvement
Pachocki also expressed concerns over recursive self-improvement, where AI models improve on their own.
He wrote, "I have a strong expectation that this speed of progress could be sustained into recursive self-improvement."
This is because the intelligence of these systems is still hard to understand.
"We may be used to thinking of AI as tools, but some agents will be pursuing their own objectives. They'll find ways to collaborate with people, by bargaining, tricking, or blackmailing them," he added.
Alignment issue
Alignment in advanced AI models
Pachocki stressed the importance of alignment in advanced AI models, saying it's the core problem in AI research and that "no lab has solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed for much longer."
He divided it into two parts: goal alignment and value alignment.
Goal alignment is whether model tries to accomplish its set goal, while value alignment is its ability to generalize from high-level principles and act reasonably in unclear/conflicting settings.
Cybersecurity concerns
Call for international cooperation and safety measures
Pachocki warned that the risk of AI is only going to increase.
He said, "A very capable agent explicitly trained and instructed to carry out nefarious acts presents a new kind of danger."
The line between abuse and misaligned autonomous actions will blur as AI gains more agency.
He also called for AI labs to voluntarily slow down frontier AI development until shared safety bars are established, and for international cooperation on future developments to be prioritized by governments worldwide.