Cases9 cases

2 to 6 of 9

1. IL-0227 Most Heroku dynos in Private Spaces lost outbound connectivity after an automated OS update restarted host networking

   Heroku2025

   Symptoms

   From 06:00 UTC on 10 June 2025, Heroku services degraded badly: customers reported intermittent logins, applications failed, and Heroku's monitoring detected widespread errors with dyno networking. Heroku's own tools and the Heroku Status Page were affected as well, which delayed communication with customers and left many of them without accurate status updates. Only at 13:58 UTC did the team find a workaround to post updates to the @herokustatus account on X.

   Network unreachableService outageHigh error rate

   Trigger

   An automated operating system update ran on Heroku's production infrastructure when it should have been disabled, and it restarted the hosts' networking services.

   Root cause

   Breaking change in an upgrade

   Component

   Host routes and routing rules

   Contributing factor

   Unpinned versionCircular dependencyLimited debug access

   Detection

   Customers reported intermittent logins, and Heroku's monitoring detected widespread errors with dyno networking. Heroku's own tools and the Heroku Status Page were affected as well.

   Remediation

   • Restarted internal instances in rotation, which gave some relief
   • Had an upstream vendor invalidate the token used for the automated updates
   • Disabled the automated upgrade service
   • Recycled dynos across the whole fleet

   Action items

   • Put permanent controls in place so no system change happens outside the controlled deployment process
   • Audit all base images for similar risks
   • Make network routing survive graceful restarts of the networking service
   • Build fully independent backup communication channels
   • Overhaul incident response tooling, including tools that run queries across the whole fleet
   • Streamline break-glass procedures for rapid access to critical systems
   • Improve monitoring to detect complex issues earlier

   Lessons

   Heroku's post-mortem names three areas: production lacked immutability controls, so an automated process could change it; communication was too slow, made worse by a status page that went down with the platform; and gaps in tooling and process made recovery take longer than it should have.

   View case
2. IL-0509 SourceHut was down for 170 hours after a DDoS led its upstream network to null-route the PHL datacenter's subnet

   SourceHut2024

   Symptoms

   At about 06:00 UTC on 10 January 2024, SourceHut's monitoring raised the alarm as a layer 3 DDoS attack hit its infrastructure in the PHL datacenter, and at about 09:00 UTC SourceHut lost all access to its PHL network. PHL was the primary datacenter, and the network became unreachable for SourceHut staff and the general public alike.

   Network unreachableService outage

   Trigger

   A layer 3 DDoS attack against SourceHut's PHL infrastructure that began at about 06:00 UTC on 10 January 2024 and broadened to more of the PHL subnet on 11 January.

   Root cause

   Dependency failure

   Component

   ISP and transit networksBGP routing

   Contributing factor

   Single point of failureLimited debug accessUntested backup

   Detection

   SourceHut's monitoring systems noticed that something was wrong and raised the alarm from about 06:00 UTC on 10 January.

   Remediation

   • Got Cogent, through the colocation provider, to narrow the null route from the /24 to the targeted /32, which mostly restored service on 10 January
   • Moved DNS to an external DNS provider
   • Brought up a new primary database in AMS
   • Restored backups from FRE to AMS
   • Applied the difference between production data and the last backups by hand
   • Provisioned a new mail system and configured the mail services for it
   • Set up an OVH box that NATs traffic through OVH's DDoS-resistant network to the production subnet in AMS
   • Used a temporary subnet from the PHL network provider for out-of-band access to the PHL servers
   • Served pages.sr.ht through an out-of-band link to PHL while its object storage was migrated
   • Emailed pages.sr.ht users whose apex records pointed at the PHL subnet with instructions
   • Ran build workers on dedicated bare-metal OVH servers

   Action items

   • Provision a new standby database
   • Configure backups from AMS to FRE
   • Install a new monitoring system
   • Retire the PHL datacenter installation and ship its servers to AMS
   • Move build service back onto SourceHut-operated servers once the build servers are shipped from PHL to AMS
   • Research and implement a way to run SourceHut in an on-premise Kubernetes cluster
   • Plan a second production-ready datacenter to load balance against, able to take over full service in a datacenter outage

   Lessons

   The operational plans SourceHut kept for failures up to the loss of a whole datacenter proved important in this incident. The incident exposed a known shortcoming in redundancy and resilience that the team was working on but had not finished, and the response of the upstream network to the attack did more damage than anything else.

   View case
3. IL-0525 OpenStreetMap.org went offline for 7h 27m when its uplink switch lost power during a data-centre power feed maintenance

   OpenStreetMap Foundation2022

   Symptoms

   From 20:18 UTC on 11 June 2022, OpenStreetMap.org was unavailable: all of its services hosted in Amsterdam, including www, the API, the wiki and planet, had no connectivity to the Internet. The outage began 18 minutes into Equinix's planned maintenance of power feed A at the Amsterdam data centre.

   Network unreachableService outage

   Trigger

   Equinix's planned 5-year maintenance of power feed A in Amsterdam, which started at 20:00 UTC on 11 June 2022.

   Root cause

   Hardware failure

   Component

   Network switchData center power and cooling

   Contributing factor

   Single point of failureLimited debug accessMonitoring blind spot

   Remediation

   • Waited for the end of the Equinix power feed A maintenance window instead of failing over to Dublin or moving the uplink; connectivity returned at 03:45 UTC

   Action items

   • Acknowledge ongoing outages faster
   • Add monitoring and alerting of switch PSU and FAN states
   • Replace the Cisco SG550X-48 switches and the Redundant Power System with Juniper EX4300 switches with redundant power supplies
   • Install out-of-band access equipment with dual power feeds in Amsterdam
   • Install redundant uplinks
   • Make site failover easier, less manual and lower risk
   • Introduce an on-call pager system for the operations SRE
   • Expand the SRE team
   • Continue and formalise fail-over tests for power, network, upstream link and service outages

   View case
4. IL-0221 A backbone maintenance command disconnected all Facebook data centers, and its DNS servers withdrew their BGP routes

   Meta2021

   Symptoms

   On 4 October 2021 Facebook's platforms went down worldwide: every connection in Facebook's global backbone network was lost, and the data centers were cut off from each other and from the internet. The rest of the internet could no longer find Facebook's servers, because Facebook's authoritative DNS servers had become unreachable.

   Service outageDNS resolution failuresNetwork unreachable

   Trigger

   During routine maintenance of the backbone, a command was issued to assess the availability of global backbone capacity, and it unintentionally took down all connections in the backbone network.

   Root cause

   Operator error

   Component

   Backbone network (WAN)BGP routingAuthoritative DNS servers and records

   Contributing factor

   Automation amplificationCircular dependencyLimited debug access

   Remediation

   • Sent engineers onsite to the data centers to debug the issue and restart systems
   • Brought the backbone network back online across the data center regions
   • Brought services back gradually, managing the rising load to avoid crashes from a traffic surge

   Action items

   • Find ways to simulate the global backbone being taken offline in storm drills
   • Strengthen testing, drills and overall resilience

   Lessons

   The hardening that protects Facebook's data centers against unauthorized access slowed recovery from an outage caused by Facebook's own error, and the author judges that tradeoff worth keeping. Storm drills prepared the teams to bring load back safely, although no drill had yet simulated the global backbone going offline.

   View case
5. IL-0222 Google Cloud US regions had heavy packet loss and some went unreachable after the network control plane was descheduled

   Google Cloud2019

   Symptoms

   On Sunday 2 June 2019, with user impact beginning between 11:47 and 11:49 US/Pacific, Google Cloud projects in several US regions saw elevated packet loss from network congestion, and some regions became unreachable. Customers saw higher latency, intermittent errors and lost connectivity to instances in us-central1, us-east1, us-east4, us-west2, northamerica-northeast1 and southamerica-east1, while us-west1, Europe and Asia had no regional congestion.

   Packet lossNetwork unreachableHigh latencyHigh error rateMonitoring outage

   Trigger

   At 11:45 US/Pacific a relatively rare type of datacenter maintenance event started in one physical location, and the automation that runs it descheduled logical clusters that also held network control plane jobs in other physical locations.

   Root cause

   Software defect

   Component

   Network control plane (SDN)BGP routingBackbone network (WAN)

   Contributing factor

   Excessive blast radiusCircular dependencyLimited debug accessIneffective rollback

   Detection

   Google engineers were alerted to the failure two minutes after it began and started the incident process used for the most serious production incidents. The first customer notice appeared on the status page at 12:53 US/Pacific.

   Remediation

   • Lowered the priority of less critical network traffic classes to reduce congestion
   • Halted the datacenter automation software responsible for the maintenance event
   • Rescheduled the network control plane and its supporting infrastructure in each location
   • Rebuilt and redistributed the lost network control plane configuration, rolling it out from 14:03
   • Directed service traffic away from the affected regions
   • Moved the Cloud VPN control plane in three regions out of its fail-safe state

   Action items

   • Keep the maintenance automation off until safeguards stop it descheduling jobs in several physical locations at once
   • Harden the cluster management software to reject such requests whatever their origin
   • Reconfigure the network control plane to reject maintenance requests of the type involved
   • Make the network control plane in each location persist its configuration
   • Let the network stay in fail static mode longer when the control plane is lost
   • Review, update and test emergency response tooling and procedures, including customer communication tooling, for such network failures
   • Extend continuous disaster recovery testing to this and similarly catastrophic failures

   View case

By root cause7