Cases3 cases

1 to 3 of 3

1. IL-0271 An empty etcd service entry crashed Discord's Elixir chat nodes, and Discord was down for most users for an hour

   Discord2020sev1

   Symptoms

   On 16 March 2020 at 14:19 PDT most of Discord's Elixir services, which handle real-time connections and chat message processing, crashed at once. Discord lost much of its real-time capacity, the failure cascaded into dependent systems, and most users could not connect for about an hour.

   Service outageHigh error rate

   Trigger

   At 14:18 the Google Cloud managed instance group autoscaler added a node to Discord's API cluster, and the TCP connection of its first announce request to etcd was reset after the request headers were sent but before the body.

   Root cause

   Software defect

   Component

   Coordination and key-value storeService discovery

   Contributing factor

   Automation amplification

   Detection

   At 14:21 several internal alarms fired on a drop in key metrics, along with anomaly alerts for cluster utilization, and a SEV1 incident was declared.

   Remediation

   • Restarted the lost real-time capacity and repaired the failed nodes
   • Throttled connections to Discord to help services recover
   • Stopped anti-entropy repairs on two ScyllaDB clusters so reconnecting users would not starve them of resources
   • Raised the connection throttle step by step to let more users reconnect

   Action items

   • Make the etcd clients send each announce request in a single TCP packet, so headers and body arrive together or not at all
   • Add error handling that ignores services whose key value is corrupt instead of crash-looping
   • File an upstream bug report with the Go project about net/http not checking the body against Content-Length

   Lessons

   Discord's service discovery code handled many etcd failure modes but did not expect a client to write a key without a value because of a transient network error. Go's net/http accepts a request whose body is shorter than its Content-Length header, which breaks the HTTP 1.1 specification and added a failure case nobody had anticipated.

   View case
2. IL-0277 Patching Elastic Cloud's ZooKeeper ensemble in us-east-1 lost quorum, and hosted clusters and Kibana went unreachable

   Elastic2019

   Symptoms

   On 4 February 2019 at about 02:50 UTC, Elastic Cloud customers with deployments in AWS us-east-1 lost stable access to their Elasticsearch clusters, and an increase in HTTP 5xx errors showed on the load balancer. Deployments were partially or completely unavailable until 09:28 UTC, Kibana access was disrupted for most customers in the region, and the Elastic Cloud User Console timed out when customers tried to view their deployments.

   High error rateService outageIntermittent timeoutsNode kernel panic

   Trigger

   A routine patching procedure in AWS us-east-1 replaced the first and second members of the three-node ZooKeeper ensemble with patched hosts between 02:25 and 02:45 UTC, following the same documented process used in the other 12 regions.

   Root cause

   Software defect

   Component

   Coordination and key-value storeContainer runtimeLinux kernelElasticsearch or OpenSearchconntrack table

   Contributing factor

   Monitoring blind spotRetry stormHealth check tied to a dependencyOverlapping failuresHarmful manual action

   Detection

   An increase in HTTP 5xx errors on the load balancer was identified at 02:50 UTC, the time the ZooKeeper ensemble lost its leader.

   Remediation

   • Restarted internal services in sweeping restarts at 03:22 and 06:00 to reduce client connection problems
   • Isolated the ZooKeeper ensemble by cutting off client connections
   • Rolled back the remaining ensemble member after its host became unstable
   • Rolled back the second ensemble member after its host became unstable
   • Rebooted the ZooKeeper hosts made unresponsive by the runc CPU softlocks
   • Re-established ZooKeeper quorum by removing client load and operator debugging, then reintroduced client load gradually
   • Increased the initLimit setting of the ZooKeeper observer layer used by the proxies
   • Scaled the proxy fleet up and out to handle the increased traffic
   • Applied a runbook that restarted the internal proxying containers and Kibana instances and set sysctl limits

   Action items

   • Reduce the dataset size carried by the ZooKeeper ensembles by over 60%
   • Reduce the number of health checks and intervals needed before a healthy proxy node joins the load balancing layer
   • Stop counting ZooKeeper connectivity in the proxy health checks once a proxy has its configuration and routes
   • Canary the rewritten proxy v2 architecture, which decouples the proxy service from ZooKeeper state, in production regions
   • Improve visibility into ZooKeeper with better logging and metrics for the ZooKeeper processes, the JVM and the container and host systems
   • Make Kibana route discovery more resilient
   • Fix the Kibana issues that amplify internal HTTP requests during availability events
   • Fix the Kibana issues that leak connections
   • Make internal monitoring agents use backoff with jitter
   • Formalize a maintenance matrix with set timeframes per region, so failures of critical services fall in non-critical times
   • Give runbooks for coordination layer maintenance a better decision criteria matrix
   • Set up a feedback loop with support leadership for incident communications through support and the status page

   Lessons

   Service metrics that reported the ZooKeeper hosts healthy were not enough to show the state of individual hosts or of the ensemble, so they gave a false signal that maintenance could continue. Because the platform depended on the coordination layer, losing its quorum caused a major service disruption in which every Elasticsearch cluster in the region saw some failures, and fixes that worked one by one fell short at the scale of the failure.

   View case
3. IL-0536 Amazon SimpleDB in US East returned HTTP 500 to nearly all API calls for two hours after its nodes left the cluster

   Amazon Web Services2014

   Symptoms

   From 9:16 PM to 11:16 PM PDT on June 13, Amazon SimpleDB in the US East Region was unavailable to all API calls except a fraction of eventually consistent reads, and SimpleDB API requests returned server-side errors (http response code 500).

   Service outageHigh error rate

   Trigger

   Power was lost to the servers that hosted several SimpleDB storage nodes in a single data center, and those storage nodes became unavailable at the same time.

   Root cause

   Software defect

   Component

   DatabaseCoordination and key-value storeData center power and cooling

   Contributing factor

   Automation amplification

   Remediation

   • Raised the lock service handshake timeout values by hand
   • Restarted a subset of metadata nodes, which could then authorize the storage nodes to rejoin the cluster
   • Throttled CreateDomain and DeleteDomain calls until 1:30 AM while the remaining metadata nodes recovered

   Action items

   • Set a longer lock service handshake timeout in all SimpleDB Regions
   • Make nodes wait and retry handshakes later with a longer timeout instead of removing themselves from the cluster after repeated timeouts

   Lessons

   The short handshake timeout had been chosen to detect replica failures quickly, and AWS concluded it was too low. Nodes removing themselves from the cluster right after several handshake timeouts widened the event and caused the API errors; the nodes should have waited and retried with a longer timeout.

   View case

By root cause1