Cases8 cases

3 to 7 of 8

1. IL-0342 DigitalOcean's AMS3, LON1 and SFO2 data centers lost Internet connectivity in repeated waves as edge routers failed

   DigitalOcean2023

   Symptoms

   From 09:55 UTC on 9 August 2023, DigitalOcean's AMS3, LON1 and SFO2 datacenters lost Internet access in waves of 10 to 15 minutes, about an hour apart. Users saw connectivity issues, latency and timeout errors on Droplets and on Droplet-based services such as Load Balancers, Kubernetes and Database clusters.

   Network unreachableIntermittent timeoutsHigh latency

   Trigger

   The edge routers received an RPKI-related routing update, and the operating system software on them crashed its routing process on that update.

   Root cause

   Software defect

   Component

   BGP routing

   Contributing factor

   Excessive blast radiusHarmful manual action

   Remediation

   • Rolled out a software update to the edge routers running the affected version, site by site
   • Rebalanced traffic on the congested routes between AMS3 and FRA1
   • Rolled out the fix to the redundant networking equipment so a failover would not hit the bug

   Action items

   • Upgrade all global edge routers to remove the bug
   • Bring the network upgrades already done in Asia to Europe and North America

   View case
2. IL-0225 An IGP purge command made every Microsoft WAN router recompute routes, causing global latency, timeouts and packet loss

   Microsoft Azure2023

   Symptoms

   From 07:08 UTC on 25 January 2023, customers saw long network latency and timeouts when connecting to resources in Azure regions and to other Microsoft services such as Microsoft 365 and Power Platform. Microsoft's monitoring detected DNS and WAN issues from 07:11 UTC.

   High latencyIntermittent timeoutsIntermittent connection failuresPacket lossNetwork unreachable

   Trigger

   A network engineer adding capacity to the global WAN in Madrid ran a step of a recently changed standard operating procedure that purged the IGP database; on the router manufacturer involved, that command runs on all IGP-joined routers instead of only the local one.

   Root cause

   Configuration change

   Component

   Backbone network (WAN)Interior routing (IGP)BGP routing

   Contributing factor

   No configuration testsUnverified assumptionExcessive blast radiusHarmful manual action

   Detection

   Monitoring detected DNS and WAN issues from 07:11 UTC and alerted the on-call engineers within 5 minutes of the command being run. The team reviewed all recent changes and identified the command by 08:20 UTC. The engineer making the change was not informed of the alerts because of the unqualified changes to the SOP.

   Remediation

   • Restarted by hand the paused WAN health and traffic engineering systems, which ended the remaining packet loss

   Action items

   • Audit and block similar widespread-impact commands across all three vendors for all WAN router roles
   • Give on-call engineers real-time visibility of approved and unqualified activity on network devices
   • Run regular mandatory operational training and attestation of following all SOPs
   • Have the Change Advisory Board review all SOPs still pending qualification within 30 days

   View case
3. IL-0241 A database shard could not elect a new primary, and the Stripe API failed most requests in two periods on one day

   Stripe2019

   Symptoms

   On 10 July 2019 the Stripe API failed a substantial majority of requests twice, from 16:35 to 17:02 UTC and from 21:14 to 22:47 UTC. In the first period automated monitoring detected within a minute that a database cluster had failed to elect a new primary.

   High error rateFailover failureWrites rejectedIntermittent timeoutsSustained high node CPU

   Trigger

   The primary node of a database cluster failed at 16:35 UTC while one shard had several stalled nodes. The second period was set off by rolling the election protocol back to the previous minor version at 20:42 UTC.

   Root cause

   Software defect

   Component

   DatabaseDatabase failover manager

   Contributing factor

   Change with a delayed effectMonitoring blind spotExcessive blast radiusHarmful manual actionUnverified assumptionOverlapping failures

   Detection

   Automated monitoring detected the failed election within a minute, and the team was alerted at 16:36 UTC. The second period was announced at 21:14 UTC by automated alerts that some shards in the cluster were unavailable.

   Remediation

   • Restarted all nodes of the database cluster, which forced a successful primary election
   • Updated the production configuration and restarted the database cluster's nodes
   • Ramped traffic back up, putting services for user-initiated API requests first

   Action items

   • Alert when nodes stop reporting their replication lag
   • Alert when a shard enters a state that could trigger the election fault
   • Work with the database maintainers on a fix for the underlying fault
   • Add circuit breaking on failed operations to particular database clusters
   • Use fault isolation to contain the impact of a single failed shard
   • Limit the resources used by clients that keep retrying failed requests
   • Add procedures and tooling that make rapid configuration changes during incident response safer

   View case
4. IL-0277 Patching Elastic Cloud's ZooKeeper ensemble in us-east-1 lost quorum, and hosted clusters and Kibana went unreachable

   Elastic2019

   Symptoms

   On 4 February 2019 at about 02:50 UTC, Elastic Cloud customers with deployments in AWS us-east-1 lost stable access to their Elasticsearch clusters, and an increase in HTTP 5xx errors showed on the load balancer. Deployments were partially or completely unavailable until 09:28 UTC, Kibana access was disrupted for most customers in the region, and the Elastic Cloud User Console timed out when customers tried to view their deployments.

   High error rateService outageIntermittent timeoutsNode kernel panic

   Trigger

   A routine patching procedure in AWS us-east-1 replaced the first and second members of the three-node ZooKeeper ensemble with patched hosts between 02:25 and 02:45 UTC, following the same documented process used in the other 12 regions.

   Root cause

   Software defect

   Component

   Coordination and key-value storeContainer runtimeLinux kernelElasticsearch or OpenSearchconntrack table

   Contributing factor

   Monitoring blind spotRetry stormHealth check tied to a dependencyOverlapping failuresHarmful manual action

   Detection

   An increase in HTTP 5xx errors on the load balancer was identified at 02:50 UTC, the time the ZooKeeper ensemble lost its leader.

   Remediation

   • Restarted internal services in sweeping restarts at 03:22 and 06:00 to reduce client connection problems
   • Isolated the ZooKeeper ensemble by cutting off client connections
   • Rolled back the remaining ensemble member after its host became unstable
   • Rolled back the second ensemble member after its host became unstable
   • Rebooted the ZooKeeper hosts made unresponsive by the runc CPU softlocks
   • Re-established ZooKeeper quorum by removing client load and operator debugging, then reintroduced client load gradually
   • Increased the initLimit setting of the ZooKeeper observer layer used by the proxies
   • Scaled the proxy fleet up and out to handle the increased traffic
   • Applied a runbook that restarted the internal proxying containers and Kibana instances and set sysctl limits

   Action items

   • Reduce the dataset size carried by the ZooKeeper ensembles by over 60%
   • Reduce the number of health checks and intervals needed before a healthy proxy node joins the load balancing layer
   • Stop counting ZooKeeper connectivity in the proxy health checks once a proxy has its configuration and routes
   • Canary the rewritten proxy v2 architecture, which decouples the proxy service from ZooKeeper state, in production regions
   • Improve visibility into ZooKeeper with better logging and metrics for the ZooKeeper processes, the JVM and the container and host systems
   • Make Kibana route discovery more resilient
   • Fix the Kibana issues that amplify internal HTTP requests during availability events
   • Fix the Kibana issues that leak connections
   • Make internal monitoring agents use backoff with jitter
   • Formalize a maintenance matrix with set timeframes per region, so failures of critical services fall in non-critical times
   • Give runbooks for coordination layer maintenance a better decision criteria matrix
   • Set up a feedback loop with support leadership for incident communications through support and the status page

   Lessons

   Service metrics that reported the ZooKeeper hosts healthy were not enough to show the state of individual hosts or of the ensemble, so they gave a false signal that maintenance could continue. Because the platform depended on the coordination layer, losing its quorum caused a major service disruption in which every Elasticsearch cluster in the region saw some failures, and fixes that worked one by one fell short at the scale of the failure.

   View case
5. IL-0387 Mailgun MongoDB secondaries were overloaded by rising connections, and a rollback then moved reads onto the primary too

   Mailgun2017

   Symptoms

   On 12 January 2017 at 14:09 UTC, Mailgun engineers began responding to alerts for the click and open tracking services and the primary MongoDB clusters in one production region. Within ten minutes the engineers found that the MongoDB secondary servers, which serve most reads, were under heavy load and slow to answer, which also caused the tracking errors. The status page reported intermittent timeouts on the Control Panel, API, SMTP and tracking links.

   Database overloadIntermittent timeoutsHigh error rate

   Trigger

   A gradual increase in connections to the MongoDB secondary servers, which coincided with several MongoDB configuration changes made the day before. A rollback of those changes then also introduced, by mistake, a change that moved the API and SMTP services from the secondaries to the primary.

   Root cause

   unknown

   Component

   Database

   Contributing factor

   Harmful manual actionSingle point of failure

   Detection

   Alerts for the click and open tracking services and the primary MongoDB clusters at 14:09 UTC.

   Remediation

   • Rolled back the previous day's MongoDB configuration changes through several pull requests from 15:15 UTC
   • Resized the MongoDB servers to handle more connections
   • Deployed connection limits at the MongoDB server level
   • Limited the number of connections the services could open to the MongoDB primary
   • Redeployed the configuration that makes services prefer secondary servers for reads

   Action items

   • Run the tracking infrastructure across two independent regions
   • Limit the number of connections every service can open to the database
   • Permanently increase the size of the primary Mongo cluster and roll the change out to the other data centers
   • Let click redirection keep working when downstream services fail

   View case

By root cause5