Skip to content
Sıfırıncı Dakika
Latest

Microsoft reveals what caused the November 12 global Microsoft 365 outage

SoftwareUpdated: 2 min read

In brief

Microsoft traced the November 12 global Microsoft 365 outage, which left more than 80% of servers unresponsive, to a software defect introduced in a July update for its network gateway servers. The company said the flaw went unnoticed for over three months before increased traffic and reduced capacity triggered a cascading failure. Service was restored to all affected customers by November 14.

  • Outage traced to a software defect added to gateway servers in July
  • The flaw went undetected for more than three months
  • Lock contention exhausted server CPU and memory
  • At its worst, over 80% of servers were unresponsive
  • Service was restored to all affected customers by November 14

The defect went unnoticed for three months

Microsoft has traced the November 12 global Microsoft 365 outage to a software defect introduced by a July update for its network gateway servers. In a preliminary post-incident report published November 15, the company said the update passed testing and was deployed worldwide without reported problems, but its load testing did not account for differences in server hardware.

Microsoft executive Charles Lamanna said testing had focused on weaker hardware with fewer cores. The defect was more severe on newer machines with more cores: lock contention inside an internal Windows process exhausted server CPU and memory, leaving machines unresponsive.

Three conditions combined to trigger the failure

Three factors set off the outage:

  • Teams switched from an alternative gateway to the primary gateway, which also supports Exchange Online, increasing load.
  • About 14% of gateway server capacity was out of service due to an unrelated issue from the previous day.
  • Together, these conditions triggered the defect and caused requests to build up.

When a server lost connectivity, traffic management redirected requests to healthy endpoints, but those servers hit the same defect under added load. Within two hours the problem had spread across the gateway fleet, and at its worst more than 80% of servers were unresponsive.

Why recovery was difficult

Restoring failed machines was hard because servers were overwhelmed by waiting traffic as soon as they returned to service. Because the defect had been present for months, standard rollback mechanisms, staged deployment controls and feature flags could not quickly reverse the change. Engineers instead developed an update that returned the affected code to its previously working state, applying it to groups of servers and restarting them before gradually returning them to service.

Timeline

Microsoft's connectivity monitor detected an increase in network problems at 16:03 UTC on November 12, and the first customer support case followed 30 minutes later. By 00:30 UTC on November 13, engineers had identified connection negotiation as the main source of access failures, and they applied the mitigation at 07:20 UTC. Teams, OneDrive, SharePoint and Office apps were able to move off the faulty gateway infrastructure, but Exchange Online could not, prolonging Outlook disruptions. Microsoft said service was restored to all affected customers by November 14 and that a final review is expected.

Why it matters

Worth reading for anyone who relies on Microsoft 365 and wants to understand what actually caused the outage, and as a case study in how a single defect can cascade through a major cloud service.

Sources

Related stories