Off-By-1-Labs study finds AI fixed 26% patches in open source
A fresh study from 1Password's Off-By-1-Labs shows that AI still has a long way to go when it comes to fixing software vulnerabilities.
Researchers expected AI to solve about two-thirds of security issues, but it only managed solid fixes for suitable patches only 26% of the time across 6,080 patch attempts on six recently disclosed vulnerabilities in open source software.
Turns out, large language models (LLMs) struggle with the tricky details of cybersecurity on their own.
Majority of 6,080 AI patches failed
More than half of the 6,080 patches made by AI either didn't fix the problem or actually introduced new bugs, sometimes looking fine on the surface but hiding real issues underneath.
Keith Hoodlet, head of Off-by-1 Labs, put it simply: AI can help spot problems, but humans are still key for making good patching decisions.
The team even shared their toolkit on GitHub so others can see where AI falls short and why human oversight is still essential.