Why godfathers of AI are worried about 'intelligence explosion'
What's the story
Geoffrey Hinton and Yoshua Bengio, two leading figures in the field of artificial intelligence (AI), have warned governments to prepare for an AI "intelligence explosion." The term refers to a dramatic acceleration of AI progress. In their report, they say this could be the most significant technological development in human history. The paper was co-authored by over 20 contributors including Jack Clark and Jakub Pachocki from Anthropic and OpenAI, respectively.
Self-improvement
Recursive self-improvement is worrying
The report highlights the potential of AI systems to improve themselves without human intervention, a process called "recursive self-improvement."
This could lead to an intelligence explosion as AIs already contribute to their own technological advancement.
Once these systems reach expert-level capabilities at AI research and development (R&D), a single developer could match the productivity of millions of top human researchers.
Risks
An 'intelligence explosion' could threaten human control over AI systems
The authors of the report warn that an intelligence explosion could be the most consequential technological development in human history.
It could compress years of progress into months or less, possibly threatening human control over AI systems.
The authors also warn it could severely erode checks on power within and between states, companies, and branches of government.
Recommendations
The report urges governments to act quickly
The report urges governments to act quickly as the window for action may close once an intelligence explosion begins.
The authors recommend three policy priorities: mandating transparent progress reports on AI-related R&D with independent auditors embedded in companies, finding ways to constrain rapid AI development, and preparing for an intelligence explosion.
Both Anthropic and OpenAI have agreed to independent evaluations of their models amid safety concerns over the pace of AI development.
Automation
Software-driven intelligence explosion is possible
The report notes that Anthropic's AI now generates 80% of its own code, while OpenAI employs autonomous AI agents for new model training.
The authors warn preliminary evidence indicates a software-driven intelligence explosion is possible.
This could lead to the extremely rapid development of highly capable or superhuman AI systems, bringing both opportunities and risks such as biological/cyber threats outpacing countermeasures and loss of human control over systems.