CrowdStrike update previously crippled Linux systems, but it went unnoticed
What's the story
Cybersecurity service provider CrowdStrike has been at the center of a global disruption, due to a problematic software update.
The faulty update caused widespread Blue Screen of Death (BSOD) issues on Windows PCs, impacting various sectors including banks, airlines, and healthcare providers.
Interestingly, CrowdStrike confirmed that this crash did not affect Linux PCs as well as Macs.
Despite the company's confirmation, it is important to note that similar problems have been occurring for months on Debian and Rocky Linux systems.
System crashes
Debian systems crashed and refused to boot
In April, a CrowdStrike update caused all Debian Linux servers in a civic tech lab to crash simultaneously and refuse to boot.
The update was incompatible with the latest stable version of Debian, despite this specific Linux configuration being supposedly supported.
After acknowledging the issue a day later, it took weeks for CrowdStrike to offer a root cause analysis.
The analysis showed that the Debian Linux configuration was not included in their test matrix.
Update concerns
Inadequate testing and compatibility issues
Rocky Linux users also reported similar problems after upgrading to RockyLinux 9.4, due to a kernel bug caused by a CrowdStrike update.
CrowdStrike support acknowledged the issue, highlighting a pattern of inadequate testing and insufficient focus on compatibility issues across various operating systems.
The disruptions in Linux systems occurred without much awareness, which raises serious concerns about CrowdStrike's software update and testing procedures.
Outage impact
About recent outage caused by CrowdStrike's update
The recent massive overnight outage affecting Windows computers worldwide was caused by a "defect" in an update for CrowdStrike's flagship security product, Falcon Sensor.
This defect caused any Windows computers with Falcon installed to crash without fully loading.
"The issue has been identified, isolated and a fix has been deployed," said CrowdStrike in a statement.
The outages affected systems including cash registers at grocery stores, departure boards at airports, work-issued laptops/desktops, airport check-in systems, airlines' ticketing/scheduling platforms, and healthcare networks.
Government involvement
Federal government responds to issues
The US federal government has been briefed on the CrowdStrike outage, and is in touch with CrowdStrike and other impacted entities.
Several federal agencies were affected by the incident, including the Department of Education and the Social Security Administration.
Homeland Security said it was working with its US cybersecurity agency CISA, CrowdStrike, and Microsoft to "fully assess and address system outages."
Solution provided
Patch and workaround for faulty update
To fix the issue, CrowdStrike has launched a patch and detailed a workaround, that could help affected systems function normally until a permanent solution is found.
The company advised users to boot their computers into Safe Mode or Windows Recovery Environment, navigate to the CrowdStrike directory, and delete the faulty file "C-00000291_.sys."
However, this manual fix could pose a big challenge for companies and organizations with large numbers of computers/Windows-powered servers in remote locations.