Over 80% of Microsoft's servers went down during global outage
What's the story
Microsoft has confirmed that more than 80% of the servers behind its networking gateway were down during last week's global outage. The company has attributed the disruption to a hidden software defect introduced in July. The outage mainly impacted Microsoft 365 services that connect with Exchange Online, Microsoft's cloud-based email service. Outlook users were unable to access their mailboxes via the web or desktop apps, and emails were delayed.
Technical details
Hidden defect caused servers to become unresponsive
Charles Lamanna, Microsoft's Executive Vice President of Copilot, Agents and Platform, explained that a hidden defect led to servers becoming overwhelmed and unresponsive under certain load conditions.
The issue stemmed from a fix specifically designed for Microsoft 365 networking gateway servers.
It went unnoticed during the company's load testing, as the tests were conducted on weaker hardware while the defect was more pronounced on newer machines with more processing cores.
Contributing factors
Change in teams' gateway triggered the outage
Notably, about 15 hours before the outage, Microsoft Teams switched from an alternative gateway to the primary one that also supports Exchange Online.
This change increased the load on that gateway and altered its traffic patterns.
At the same time, nearly 14% of the gateway's server capacity was out of service due to an unrelated issue from the previous day.
Bug activation
Backlog of requests during peak traffic hours in US
The combination of the increased load and reduced capacity triggered the latent bug from July, resulting in a backlog of requests until servers stopped responding.
This incident occurred during peak traffic hours in the United States.
Lamanna explained that when servers lose connectivity, their traffic management systems automatically reroute users to other healthy endpoints to prevent any disruption due to hardware, software, or network defects.
Cascading failure
Cascading failure across Microsoft's fleet of servers
The redirected traffic increased the likelihood of the same problem occurring on other endpoints.
This led to a cascading failure from server to server as the same amount of load was spread across fewer servers.
In less than two hours, the previously unknown defect was encountered across Microsoft's fleet and rendered them unresponsive.