Cases9 cases

4 to 8 of 9

1. IL-0525 OpenStreetMap.org went offline for 7h 27m when its uplink switch lost power during a data-centre power feed maintenance

   OpenStreetMap Foundation2022

   Symptoms

   From 20:18 UTC on 11 June 2022, OpenStreetMap.org was unavailable: all of its services hosted in Amsterdam, including www, the API, the wiki and planet, had no connectivity to the Internet. The outage began 18 minutes into Equinix's planned maintenance of power feed A at the Amsterdam data centre.

   Network unreachableService outage

   Trigger

   Equinix's planned 5-year maintenance of power feed A in Amsterdam, which started at 20:00 UTC on 11 June 2022.

   Root cause

   Hardware failure

   Component

   Network switchData center power and cooling

   Contributing factor

   Single point of failureLimited debug accessMonitoring blind spot

   Remediation

   • Waited for the end of the Equinix power feed A maintenance window instead of failing over to Dublin or moving the uplink; connectivity returned at 03:45 UTC

   Action items

   • Acknowledge ongoing outages faster
   • Add monitoring and alerting of switch PSU and FAN states
   • Replace the Cisco SG550X-48 switches and the Redundant Power System with Juniper EX4300 switches with redundant power supplies
   • Install out-of-band access equipment with dual power feeds in Amsterdam
   • Install redundant uplinks
   • Make site failover easier, less manual and lower risk
   • Introduce an on-call pager system for the operations SRE
   • Expand the SRE team
   • Continue and formalise fail-over tests for power, network, upstream link and service outages

   View case
2. IL-0221 A backbone maintenance command disconnected all Facebook data centers, and its DNS servers withdrew their BGP routes

   Meta2021

   Symptoms

   On 4 October 2021 Facebook's platforms went down worldwide: every connection in Facebook's global backbone network was lost, and the data centers were cut off from each other and from the internet. The rest of the internet could no longer find Facebook's servers, because Facebook's authoritative DNS servers had become unreachable.

   Service outageDNS resolution failuresNetwork unreachable

   Trigger

   During routine maintenance of the backbone, a command was issued to assess the availability of global backbone capacity, and it unintentionally took down all connections in the backbone network.

   Root cause

   Operator error

   Component

   Backbone network (WAN)BGP routingAuthoritative DNS servers and records

   Contributing factor

   Automation amplificationCircular dependencyLimited debug access

   Remediation

   • Sent engineers onsite to the data centers to debug the issue and restart systems
   • Brought the backbone network back online across the data center regions
   • Brought services back gradually, managing the rising load to avoid crashes from a traffic surge

   Action items

   • Find ways to simulate the global backbone being taken offline in storm drills
   • Strengthen testing, drills and overall resilience

   Lessons

   The hardening that protects Facebook's data centers against unauthorized access slowed recovery from an outage caused by Facebook's own error, and the author judges that tradeoff worth keeping. Storm drills prepared the teams to bring load back safely, although no drill had yet simulated the global backbone going offline.

   View case
3. IL-0222 Google Cloud US regions had heavy packet loss and some went unreachable after the network control plane was descheduled

   Google Cloud2019

   Symptoms

   On Sunday 2 June 2019, with user impact beginning between 11:47 and 11:49 US/Pacific, Google Cloud projects in several US regions saw elevated packet loss from network congestion, and some regions became unreachable. Customers saw higher latency, intermittent errors and lost connectivity to instances in us-central1, us-east1, us-east4, us-west2, northamerica-northeast1 and southamerica-east1, while us-west1, Europe and Asia had no regional congestion.

   Packet lossNetwork unreachableHigh latencyHigh error rateMonitoring outage

   Trigger

   At 11:45 US/Pacific a relatively rare type of datacenter maintenance event started in one physical location, and the automation that runs it descheduled logical clusters that also held network control plane jobs in other physical locations.

   Root cause

   Software defect

   Component

   Network control plane (SDN)BGP routingBackbone network (WAN)

   Contributing factor

   Excessive blast radiusCircular dependencyLimited debug accessIneffective rollback

   Detection

   Google engineers were alerted to the failure two minutes after it began and started the incident process used for the most serious production incidents. The first customer notice appeared on the status page at 12:53 US/Pacific.

   Remediation

   • Lowered the priority of less critical network traffic classes to reduce congestion
   • Halted the datacenter automation software responsible for the maintenance event
   • Rescheduled the network control plane and its supporting infrastructure in each location
   • Rebuilt and redistributed the lost network control plane configuration, rolling it out from 14:03
   • Directed service traffic away from the affected regions
   • Moved the Cloud VPN control plane in three regions out of its fail-safe state

   Action items

   • Keep the maintenance automation off until safeguards stop it descheduling jobs in several physical locations at once
   • Harden the cluster management software to reject such requests whatever their origin
   • Reconfigure the network control plane to reject maintenance requests of the type involved
   • Make the network control plane in each location persist its configuration
   • Let the network stay in fail static mode longer when the control plane is lost
   • Review, update and test emergency response tooling and procedures, including customer communication tooling, for such network failures
   • Extend continuous disaster recovery testing to this and similarly catastrophic failures

   View case
4. IL-0261 Discord went fully offline twice as short network losses on Google Cloud sessions nodes wedged Erlang distribution

   Discord2017

   Symptoms

   On 17 and 18 November 2017 nodes of Discord's 'sessions' cluster on Google Cloud Platform lost network connectivity for short periods, 16 times in all. Engineers saw a service impact, a node whose service had died and restarted, and pages for low memory on other sessions nodes; twice the failure cascaded and Discord was completely offline for about 30 minutes.

   Network unreachableMemory pressureService outage

   Trigger

   A software change that Google rolled out to the Google Cloud Platform networking stack on Friday introduced a bug that made VMs in Discord's sessions cluster lose network connectivity for short periods.

   Root cause

   Cloud provider outage

   Component

   Service discovery

   Contributing factor

   Limited debug accessHarmful manual action

   Detection

   At 11:52 PST on 17 November engineers noticed a service impact and escalated it in the incident chat. Later affected nodes showed up as pages to the on-call engineer for low memory on sessions nodes, and on 18 November as a page for a networking incident on sessions-1-11 together with issue reports that the customer experience team escalated from support channels.

   Remediation

   • Forcefully terminated sessions nodes whose Erlang distribution port had failed
   • Escalated the Google support ticket to P1 and notified the Technical Account Manager
   • Restarted several core Discord services on 17 November, making clients reconnect over about 20 minutes
   • Began terminating and rebooting the entire sessions cluster after seeing that rebooted machines did not fail again
   • Disabled message sending globally on 18 November to help the service recover
   • Restarted all members of the sessions cluster
   • Re-enabled message sending after the initial all clear
   • Google rolled back the software change made to its networking stack

   Action items

   • Continue the project to improve how Discord's Erlang clusters handle node failure
   • Expand how upgrades to these services are load tested so the rollout can be completed safely
   • Replace the HTTP library used by Discord's Elixir/Erlang services
   • Fix the instability seen in Erlang's net_kernel component
   • Replace the pieces of the service discovery component that degraded with more performant implementations
   • Work with Google on the issues Discord met while working with Google on this incident
   • Expand the internal procedure for communicating with Google so the progress of P1 issues is tracked

   Lessons

   Discord concluded that performance and stability issues in its own system, in Erlang's net_kernel, the service discovery component and the HTTP library, were what turned short network losses on part of the fleet from isolated issues for some users into two full system outages.

   View case
5. IL-0494 Netlify's central load balancer IP stopped responding when Rackspace null-routed it in a DDoS, and A-record sites hung

   Netlify2016

   Symptoms

   At 8am PST on Monday, August 22, 2016, Netlify got alerts that its central load balancer, the single public IP that users point A records at, had stopped responding. Rackspace had null-routed the load balancer, so all traffic to that IP hung before it reached Netlify's servers.

   Network unreachable

   Trigger

   A DDoS attack against Netlify's central load balancer in Rackspace's Chicago datacenter, after which Rackspace null-routed the load balancer's public IP.

   Root cause

   Cloud provider outage

   Component

   Cloud load balancer

   Contributing factor

   Single point of failureLimited debug access

   Detection

   Alerts at 8am PST reported that the central load balancer had stopped responding, and Rackspace then confirmed it had null-routed the load balancer.

   Remediation

   • Started a new cluster of CDN machines in Google Compute behind a Google network load balancer
   • Gave clients who had contacted support the new public IP and told them to change their A records
   • Updated the documentation and announced the new IP on Twitter, Gitter and the status page

   Action items

   • Keep the new Google-based network IP running in place of the Rackspace load balancer
   • Release Netlify's own DNS hosting, integrated with the CDN, so domains do not depend on a fixed IP
   • Fix the status page publishing process so it is updated straight away during an issue
   • Build a mailer that sends targeted mails to all paying clients during an outage

   Lessons

   Netlify judged Rackspace's load balancer too vulnerable to DDoS and Rackspace's DDoS processes not up to the standard of other cloud providers, and expected Google's anycast-based Maglev load balancer to be more resilient. In the end, the only way to protect a domain from DDoS-related outages is not to expose any fixed IP address for it, which is why sites should use the www domain with a CNAME, or Netlify DNS, rather than a single A-record IP.

   View case

By root cause7