AI whistleblowers warn of safety risks at NYC Council hearing
What's the story
AI industry insiders sounded the alarm before New York City lawmakers on Monday, urging tighter oversight amid mounting fears that advanced systems could pose existential risks to humanity. Representatives from OpenAI, Meta, Google, and Anthropic all participated in this important discussion. They emphasized the need for robust safety measures to prevent potential harm from advanced AI systems.
Oversight demand
Reckless culture at big tech firms, ex-employees claim
At the hearing, former employees of Anthropic, OpenAI, and Google DeepMind spoke about a reckless culture within these companies.
They claimed that major tech firms are racing to develop technologies they don't fully understand how to control.
Jacob Coxon, an ex-Anthropic engineer who now advocates for regulation, said the companies run on a start-up mindset: Move fast, break things, fix them later.
Risk alert
Coxon's warning about AI's potential dangers
Coxon, who recently left Anthropic, said that while AI has great potential, without transparency and safeguards, it could lead to disastrous outcomes.
"On the current path, I think it is more likely than not that humanity loses control to these AIs and it could end in human extinction," he said.
His stark warning underscores the need for careful consideration of AI development.
Defense stance
Tech companies defend their AI systems during the hearing
A panel of representatives from OpenAI, Anthropic, Google, and Meta defended their AI systems at the hearing.
They said these technologies are already improving people's daily lives and emphasized their commitment to safety.
Morgan Dwyer from OpenAI said he shares the Council's commitment to ensuring technology is developed safely and its benefits are widely shared.
Risk quantification
'We shouldn't train models we can't keep under human control'
During the hearing, City Council Speaker Julie Menin asked tech company representatives to quantify the risk posed by AI in a worst-case catastrophic scenario.
Dwyer responded that he doesn't know but doesn't think it matters whether it's a 1%, 10%, or 20% chance of something going wrong.
He said none of these levels are acceptable and stressed we shouldn't train models we can't keep under human control.
Regulation proposals
NYC considering regulations around AI technology
The City Council is considering several AI regulations, including requiring companies to disclose information about their tools and seeking third-party approval for systems in the city.
They are also considering allowing whistleblowers to get a portion of fines from companies that break these laws.
Separately, New York Governor Kathy Hochul has also pushed for reporting and transparency standards around AI technology.