Cloudflare reveals reason for outage that knocked ChatGPT, X offline
What's the story
Cloudflare, a major internet infrastructure provider, has detailed the cause of Tuesday's outage that disrupted services like ChatGPT and X. The company attributed the incident to a misconfiguration in its Bot Management system, which is designed to regulate automated web crawlers on its Content Delivery Network (CDN). Cloudflare CEO Matthew Prince explained in a blog post that this was their "worst outage since 2019."
System failure
Bot management system's role in the outage
The Bot Management system, as Prince explained, is meant to prevent issues like data scraping by crawlers. However, a change in ClickHouse query behavior led to this major outage. The machine learning model behind Bot Management uses a frequently updated configuration file to identify automated requests. But an alteration in ClickHouse query behavior caused it to generate duplicate 'feature' rows, leading to service disruptions across Cloudflare's network.
Service disruption
Outage's impact on Cloudflare's services
The duplication of information in the configuration file quickly exceeded memory limits, crashing "the core proxy system that handles traffic processing for our customers." This affected any traffic relying on the bots module. Companies using Cloudflare's rules to block certain bots saw false positives and real traffic was cut off. However, those not using the generated bot score in their rules remained unaffected during this outage.
Response strategy
Cloudflare's response and future prevention measures
In light of this incident, Cloudflare has outlined four strategies to prevent similar issues in the future. These include hardening ingestion of configuration files, enabling more global kill switches for features, preventing core dumps or error reports from overwhelming system resources, and reviewing failure modes for error conditions across all core proxy modules. Despite these measures, Prince acknowledged that the growing centralization of internet services could make such outages unavoidable.