These Chinese AI models told researchers how to assassinate people
What's the story
Moonshot, a Chinese artificial intelligence (AI) developer, is conducting an internal review after two of its popular models were manipulated into providing information on how to create biological weapons and carry out assassinations. The incident was discovered by Mindgard, a company that tests the security of AI systems. It found that Kimi K2.6 and K3 Swarm could bypass safety limits set by their developers during a process called "jailbreaking."
Security breach
Jailbroken Kimi 2.6 could let hackers execute code
Jailbreaking is a method where researchers use complex instructions to check if AI tools ignore guardrails.
Mindgard said this process should have prevented Kimi from discussing sensitive topics.
The firm also warned that a jailbroken Kimi 2.6 could let hackers execute code on its computing resources and access the internet, making it a potential launchpad for cyberattacks.
Company response
Moonshot in talks with Mindgard about findings
In light of these revelations, Moonshot AI has welcomed third-party input as a key element in creating better and safer AI. The company is also in talks with Mindgard about its findings.
Industry divide
AI industry continues to be split
The findings come as the industry continues to be split over whether closed, proprietary AI like ChatGPT and Anthropic's Claude systems are safer than open-source tools.
Kimi is an open-weight model, which means it could theoretically be run on its own computing infrastructure.
While some experts fear these models could fall into the wrong hands, others believe they could also be used for cyber-defense.