Cases2 cases

1 to 2 of 2

1. IL-0221 A backbone maintenance command disconnected all Facebook data centers, and its DNS servers withdrew their BGP routes

   Meta2021

   Symptoms

   On 4 October 2021 Facebook's platforms went down worldwide: every connection in Facebook's global backbone network was lost, and the data centers were cut off from each other and from the internet. The rest of the internet could no longer find Facebook's servers, because Facebook's authoritative DNS servers had become unreachable.

   Service outageDNS resolution failuresNetwork unreachable

   Trigger

   During routine maintenance of the backbone, a command was issued to assess the availability of global backbone capacity, and it unintentionally took down all connections in the backbone network.

   Root cause

   Operator error

   Component

   Backbone network (WAN)BGP routingAuthoritative DNS servers and records

   Contributing factor

   Automation amplificationCircular dependencyLimited debug access

   Remediation

   • Sent engineers onsite to the data centers to debug the issue and restart systems
   • Brought the backbone network back online across the data center regions
   • Brought services back gradually, managing the rising load to avoid crashes from a traffic surge

   Action items

   • Find ways to simulate the global backbone being taken offline in storm drills
   • Strengthen testing, drills and overall resilience

   Lessons

   The hardening that protects Facebook's data centers against unauthorized access slowed recovery from an outage caused by Facebook's own error, and the author judges that tradeoff worth keeping. Storm drills prepared the teams to bring load back safely, although no drill had yet simulated the global backbone going offline.

   View case
2. IL-0222 Google Cloud US regions had heavy packet loss and some went unreachable after the network control plane was descheduled

   Google Cloud2019

   Symptoms

   On Sunday 2 June 2019, with user impact beginning between 11:47 and 11:49 US/Pacific, Google Cloud projects in several US regions saw elevated packet loss from network congestion, and some regions became unreachable. Customers saw higher latency, intermittent errors and lost connectivity to instances in us-central1, us-east1, us-east4, us-west2, northamerica-northeast1 and southamerica-east1, while us-west1, Europe and Asia had no regional congestion.

   Packet lossNetwork unreachableHigh latencyHigh error rateMonitoring outage

   Trigger

   At 11:45 US/Pacific a relatively rare type of datacenter maintenance event started in one physical location, and the automation that runs it descheduled logical clusters that also held network control plane jobs in other physical locations.

   Root cause

   Software defect

   Component

   Network control plane (SDN)BGP routingBackbone network (WAN)

   Contributing factor

   Excessive blast radiusCircular dependencyLimited debug accessIneffective rollback

   Detection

   Google engineers were alerted to the failure two minutes after it began and started the incident process used for the most serious production incidents. The first customer notice appeared on the status page at 12:53 US/Pacific.

   Remediation

   • Lowered the priority of less critical network traffic classes to reduce congestion
   • Halted the datacenter automation software responsible for the maintenance event
   • Rescheduled the network control plane and its supporting infrastructure in each location
   • Rebuilt and redistributed the lost network control plane configuration, rolling it out from 14:03
   • Directed service traffic away from the affected regions
   • Moved the Cloud VPN control plane in three regions out of its fail-safe state

   Action items

   • Keep the maintenance automation off until safeguards stop it descheduling jobs in several physical locations at once
   • Harden the cluster management software to reject such requests whatever their origin
   • Reconfigure the network control plane to reject maintenance requests of the type involved
   • Make the network control plane in each location persist its configuration
   • Let the network stay in fail static mode longer when the control plane is lost
   • Review, update and test emergency response tooling and procedures, including customer communication tooling, for such network failures
   • Extend continuous disaster recovery testing to this and similarly catastrophic failures

   View case

By root cause2