Off-By-1 Labs study finds AI vulnerability patches worked about 25%
Technology
A new study from Off-By-1-Labs (part of 1Password) found that artificial intelligence (AI) models like Claude and an LLM based on OpenAI's coding agent, Codex, aren't very reliable at patching software vulnerabilities.
Out of more than 6,000 fixes generated by these AIs, only about a quarter actually worked.
AI fixes often introduce new bugs
Turns out, when AI tries to fix code, it often creates new bugs or messes up how apps work.
The researchers say AI is helpful at spotting problems but not so much at solving them, at least for now.
As Keith Hoodlet, head of Off-by-1 Labs, puts it, teaming up human know-how with AI tools is the best way to keep software safe.
If you're curious, 1Password has even put its FLAWED tool on GitHub for anyone who wants to dig deeper.