Cases3 cases

1 to 3 of 3

1. IL-0509 SourceHut was down for 170 hours after a DDoS led its upstream network to null-route the PHL datacenter's subnet

   SourceHut2024

   Symptoms

   At about 06:00 UTC on 10 January 2024, SourceHut's monitoring raised the alarm as a layer 3 DDoS attack hit its infrastructure in the PHL datacenter, and at about 09:00 UTC SourceHut lost all access to its PHL network. PHL was the primary datacenter, and the network became unreachable for SourceHut staff and the general public alike.

   Network unreachableService outage

   Trigger

   A layer 3 DDoS attack against SourceHut's PHL infrastructure that began at about 06:00 UTC on 10 January 2024 and broadened to more of the PHL subnet on 11 January.

   Root cause

   Dependency failure

   Component

   ISP and transit networksBGP routing

   Contributing factor

   Single point of failureLimited debug accessUntested backup

   Detection

   SourceHut's monitoring systems noticed that something was wrong and raised the alarm from about 06:00 UTC on 10 January.

   Remediation

   • Got Cogent, through the colocation provider, to narrow the null route from the /24 to the targeted /32, which mostly restored service on 10 January
   • Moved DNS to an external DNS provider
   • Brought up a new primary database in AMS
   • Restored backups from FRE to AMS
   • Applied the difference between production data and the last backups by hand
   • Provisioned a new mail system and configured the mail services for it
   • Set up an OVH box that NATs traffic through OVH's DDoS-resistant network to the production subnet in AMS
   • Used a temporary subnet from the PHL network provider for out-of-band access to the PHL servers
   • Served pages.sr.ht through an out-of-band link to PHL while its object storage was migrated
   • Emailed pages.sr.ht users whose apex records pointed at the PHL subnet with instructions
   • Ran build workers on dedicated bare-metal OVH servers

   Action items

   • Provision a new standby database
   • Configure backups from AMS to FRE
   • Install a new monitoring system
   • Retire the PHL datacenter installation and ship its servers to AMS
   • Move build service back onto SourceHut-operated servers once the build servers are shipped from PHL to AMS
   • Research and implement a way to run SourceHut in an on-premise Kubernetes cluster
   • Plan a second production-ready datacenter to load balance against, able to take over full service in a datacenter outage

   Lessons

   The operational plans SourceHut kept for failures up to the loss of a whole datacenter proved important in this incident. The incident exposed a known shortcoming in redundancy and resilience that the team was working on but had not finished, and the response of the upstream network to the attack did more damage than anything else.

   View case
2. IL-0221 A backbone maintenance command disconnected all Facebook data centers, and its DNS servers withdrew their BGP routes

   Meta2021

   Symptoms

   On 4 October 2021 Facebook's platforms went down worldwide: every connection in Facebook's global backbone network was lost, and the data centers were cut off from each other and from the internet. The rest of the internet could no longer find Facebook's servers, because Facebook's authoritative DNS servers had become unreachable.

   Service outageDNS resolution failuresNetwork unreachable

   Trigger

   During routine maintenance of the backbone, a command was issued to assess the availability of global backbone capacity, and it unintentionally took down all connections in the backbone network.

   Root cause

   Operator error

   Component

   Backbone network (WAN)BGP routingAuthoritative DNS servers and records

   Contributing factor

   Automation amplificationCircular dependencyLimited debug access

   Remediation

   • Sent engineers onsite to the data centers to debug the issue and restart systems
   • Brought the backbone network back online across the data center regions
   • Brought services back gradually, managing the rising load to avoid crashes from a traffic surge

   Action items

   • Find ways to simulate the global backbone being taken offline in storm drills
   • Strengthen testing, drills and overall resilience

   Lessons

   The hardening that protects Facebook's data centers against unauthorized access slowed recovery from an outage caused by Facebook's own error, and the author judges that tradeoff worth keeping. Storm drills prepared the teams to bring load back safely, although no drill had yet simulated the global backbone going offline.

   View case
3. IL-0258 FastMail was unreachable for almost two hours when the GRE tunnels carrying its traffic from Level 3 went down

   Fastmail2016

   Symptoms

   At 4.50am Melbourne time on Sunday 7 February (evening of 6 February UTC), FastMail became unavailable and stayed unreachable for almost two hours. Traffic to FastMail's public IP range never made it into FastMail's own network.

   Service outageNetwork unreachable

   Trigger

   The GRE tunnels between Level 3 and NYI, which carried FastMail's filtered traffic into its network, went down.

   Root cause

   Dependency failure

   Component

   DDoS protection serviceBGP routing

   Contributing factor

   Limited debug access

   Detection

   NYI, the datacentre provider, noticed the failure quickly.

   Remediation

   • NYI escalated with Level 3's support until FastMail's public IP ranges were pointed back to NYI

   Action items

   • Go back to DDoS protection from Black Lotus under NYI's explicit control
   • Build nginx frontends in the Los Angeles datacentre that reach NYI over the secondary network, and point DNS to them if NYI's primary network fails
   • Add hardware in Amsterdam and change the delivery system so it depends less on NYI
   • Work with NYI to understand its processes for detecting and responding to network faults

   Lessons

   Much of FastMail's and NYI's reliability had come from NYI controlling its own infrastructure. With protection always on at Level 3, NYI could not restore the routes itself, and FastMail calls the resulting delay in restoring service unacceptable.

   View case

By root cause2