IL-0221 Meta · 2021-10-04networking
A backbone maintenance command disconnected all Facebook data centers, and its DNS servers withdrew their BGP routes
Facebook's backbone is the fiber network that connects all of its computing facilities: large data centers that store data and run the heavy computation, and smaller facilities that connect the backbone to the internet. Engineers often take part of the backbone offline for maintenance. In one such routine job a command meant to assess global backbone capacity took down every backbone connection. Facebook's systems audit commands like this to stop such mistakes, but a bug in the audit tool kept it from stopping this one. The smaller facilities also answer DNS queries, and their authoritative name servers withdraw their BGP advertisements when they cannot reach the data centers, so the loss of the backbone made the DNS servers unreachable even though they were working.
Normal and out-of-band network access to the data centers was down, and the total loss of DNS broke many internal tools used to investigate outages. Engineers went onsite to the data centers to debug and restart systems; activating the secure access protocols to get them in and able to work took extra time. After they confirmed the problem they brought the backbone back online, and services came back with it. Turning everything on at once risked a new round of crashes from a surge in traffic: individual data centers showed dips in power usage of tens of megawatts, and reversing such a dip suddenly could endanger electrical systems and caches. Using experience from storm drills, in which a service, data center or whole region is taken offline on purpose, the teams brought load back carefully and services returned without further systemwide failures.
- Severity
- –
- Scope
- All of Facebook's platforms: data centers disconnected from each other and from the internet, and Facebook's DNS servers unreachable from the rest of the internet
- Users affected
- –
- Region
- global
- Time to detect
- –
- Time to resolve
- –
Symptoms
On 4 October 2021 Facebook's platforms went down worldwide: every connection in Facebook's global backbone network was lost, and the data centers were cut off from each other and from the internet. The rest of the internet could no longer find Facebook's servers, because Facebook's authoritative DNS servers had become unreachable.
The DNS servers were still running. They sit in the smaller facilities that connect the backbone to the internet, and they stop advertising their well-known IP addresses over BGP when they cannot reach the data centers. With the whole backbone out of operation, every one of these locations declared itself unhealthy and withdrew its BGP advertisements.
Responders could not reach the data centers by the normal means because their networks were down, and the out-of-band network access was down as well. The total loss of DNS also broke many of the internal tools normally used to investigate and fix such outages. Engineers were sent onsite, where the physical and system security of the facilities, and routers built to be hard to modify even with physical access, made it slow to start work.
Once backbone connectivity was restored, individual data centers were reporting dips in power usage in the range of tens of megawatts. Reversing such a dip suddenly could put everything from electrical systems to caches at risk, and turning all services back on at once could cause a new round of crashes from a surge in traffic.
Symptom Service outageDNS resolution failuresNetwork unreachable
Root cause
Root cause Operator error
- Trigger
- During routine maintenance of the backbone, a command was issued to assess the availability of global backbone capacity, and it unintentionally took down all connections in the backbone network.
Detection
–
Remediation
- Sent engineers onsite to the data centers to debug the issue and restart systems
- Brought the backbone network back online across the data center regions
- Brought services back gradually, managing the rising load to avoid crashes from a traffic surge
Action items
- Find ways to simulate the global backbone being taken offline in storm drills
- Strengthen testing, drills and overall resilience
Lessons
The hardening that protects Facebook's data centers against unauthorized access slowed recovery from an outage caused by Facebook's own error, and the author judges that tradeoff worth keeping. Storm drills prepared the teams to bring load back safely, although no drill had yet simulated the global backbone going offline.
Sources
Related cases
IL-0348Scaleway's FR-PAR region lost public internet connectivity when one 100 Gbps backbone link was saturatedScaleway · 2025same root cause · shared Authoritative DNS servers and records, Backbone network (WAN)
IL-0531Azure Database for MySQL Single Server and MariaDB in West US rejected every new connection for a batch of serversMicrosoft Azure · 2024same root cause · shared Authoritative DNS servers and records
Check your cluster for this
For AI tools and scripts
- This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0221
- The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0221/similar
- MCP server (Streamable HTTP, no key):
https://incidentlib-mcp.benchlet.com/mcp
The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.