Anthropic blocks scientists using Claude for bioweapon research
What's the story
Anthropic has flagged and stopped several scientists from using its AI models to develop potential biological weapons. The company's latest report details five case studies where its models were used in such a manner. It also explains the safety measures that detected this misuse and how the company responded to it.
Detection challenges
Distinguishing legitimate research from potential misuse
Anthropic has stressed the difficulty of distinguishing between legitimate research and potential misuse. For instance, developing a new vaccine could resemble creating a bioweapon.
The company has been overly cautious in all cases due to the possible consequences of such activities.
Jacob Klein, Anthropic's head of threat intelligence, told The New York Times that these situations are nuanced and not as straightforward as someone openly expressing their intent to create a biological weapon.
Research concerns
Chikungunya virus gain-of-function research flagged by Claude
In one case from May, Anthropic's "biological safety classifier" flagged a request for Claude to prepare a grant for gain-of-function research on the chikungunya virus.
The company noted that this mosquito-borne virus has no licensed treatment and can cause severe pain and fever for weeks or months.
The proposed research sought to identify mutations affecting transmissibility and immune evasion, then engineer those changes into the virus.
Institute affiliation
Proposed research was affiliated with military institute
The fact that this proposed research was affiliated with a military research institute raised further concerns for Anthropic.
The company has also flagged other cases of gain-of-function research into bird flu and a researcher using Claude to develop an atlas of venom toxin peptides and "a generative pipeline that optimized toxin characteristics."
These instances highlight the potential misuse of AI models in sensitive biological research.
Additional misuse
Addressing the issue of AI model misuse
The report also highlights the misuse of Anthropic's models as a surveillance tool and to create software exploits, propaganda, and weapons systems.
The company has banned accounts associated with these violations and is using the findings from its investigation to improve model safeguards against future misuse.
However, given the complex nature of biological misuse, Anthropic isn't naming those involved in developing potential biological weapons with its models.