Cases3 cases

1 to 3 of 3

1. IL-0240 Cloudflare API requests failed and the dashboard slowed for hours after the authentication database lost its replicas

   Cloudflare2020

   Symptoms

   On 2 November 2020 the Cloudflare API and dashboard were degraded for six hours and 33 minutes. The success rate of API queries dropped at times to 75%, and the dashboard was up to 80 times slower than normal, while Cloudflare's edge network kept working.

   High error rateHigh latencyDatabase overloadWrites rejectedPacket loss

   Trigger

   At 14:43 UTC a network switch in a control plane data center partly failed: it kept LACP and BGP running but stopped forwarding some packets, and it recovered on its own six minutes later.

   Root cause

   DNS or network fault

   Component

   NIC, cable and switch port linkCoordination and key-value storeDatabase failover managerDatabaseDatabase replication

   Contributing factor

   Automation amplificationNo headroomHarmful manual action

   Detection

   At 14:43 UTC alerts fired because the misbehaving switch was unreachable to pings. Engineers then saw that the primary authentication database was overwhelmed.

   Remediation

   • Shed load from the primary authentication database by slowing SSL certificate pushes to the edge, emails and other features
   • Moved API traffic that could use read replicas to the secondary data center by hand
   • Sent authentication calls fully back to the primary data center when the dashboard got worse
   • Rebuilt the first database replica, which put itself back into service at 21:20 UTC and returned performance to normal
   • Ramped the turned-down services back up so asynchronous processing could catch up

   Action items

   • Remove the cluster management limitation that requires rebuilding all replicas after a primary is promoted
   • Redesign the user sessions system so dashboard traffic can be steered between data centers
   • Change the auto-remediation configuration so a database replica is not promoted so quickly
   • Tune each system involved to be more resilient to failures of its dependencies

   Lessons

   Every system in the chain had redundancy and none failed completely; each only degraded, which made the chain of events much harder to model and foresee. Automating replica promotion so that it could trigger on a minute's notice had been a point of pride, but for at least one database a fast promotion can do more harm than the failure it answers. Most general-purpose cluster managers use Raft or Paxos rather than Byzantine fault tolerant protocols, accepting a rare failure mode in exchange for a protocol that is simpler to implement correctly and debug.

   View case
2. IL-0271 An empty etcd service entry crashed Discord's Elixir chat nodes, and Discord was down for most users for an hour

   Discord2020sev1

   Symptoms

   On 16 March 2020 at 14:19 PDT most of Discord's Elixir services, which handle real-time connections and chat message processing, crashed at once. Discord lost much of its real-time capacity, the failure cascaded into dependent systems, and most users could not connect for about an hour.

   Service outageHigh error rate

   Trigger

   At 14:18 the Google Cloud managed instance group autoscaler added a node to Discord's API cluster, and the TCP connection of its first announce request to etcd was reset after the request headers were sent but before the body.

   Root cause

   Software defect

   Component

   Coordination and key-value storeService discovery

   Contributing factor

   Automation amplification

   Detection

   At 14:21 several internal alarms fired on a drop in key metrics, along with anomaly alerts for cluster utilization, and a SEV1 incident was declared.

   Remediation

   • Restarted the lost real-time capacity and repaired the failed nodes
   • Throttled connections to Discord to help services recover
   • Stopped anti-entropy repairs on two ScyllaDB clusters so reconnecting users would not starve them of resources
   • Raised the connection throttle step by step to let more users reconnect

   Action items

   • Make the etcd clients send each announce request in a single TCP packet, so headers and body arrive together or not at all
   • Add error handling that ignores services whose key value is corrupt instead of crash-looping
   • File an upstream bug report with the Go project about net/http not checking the body against Content-Length

   Lessons

   Discord's service discovery code handled many etcd failure modes but did not expect a client to write a key without a value because of a transient network error. Go's net/http accepts a request whose body is shorter than its Content-Length header, which breaks the HTTP 1.1 specification and added a failure case nobody had anticipated.

   View case
3. IL-0536 Amazon SimpleDB in US East returned HTTP 500 to nearly all API calls for two hours after its nodes left the cluster

   Amazon Web Services2014

   Symptoms

   From 9:16 PM to 11:16 PM PDT on June 13, Amazon SimpleDB in the US East Region was unavailable to all API calls except a fraction of eventually consistent reads, and SimpleDB API requests returned server-side errors (http response code 500).

   Service outageHigh error rate

   Trigger

   Power was lost to the servers that hosted several SimpleDB storage nodes in a single data center, and those storage nodes became unavailable at the same time.

   Root cause

   Software defect

   Component

   DatabaseCoordination and key-value storeData center power and cooling

   Contributing factor

   Automation amplification

   Remediation

   • Raised the lock service handshake timeout values by hand
   • Restarted a subset of metadata nodes, which could then authorize the storage nodes to rejoin the cluster
   • Throttled CreateDomain and DeleteDomain calls until 1:30 AM while the remaining metadata nodes recovered

   Action items

   • Set a longer lock service handshake timeout in all SimpleDB Regions
   • Make nodes wait and retry handshakes later with a longer timeout instead of removing themselves from the cluster after repeated timeouts

   Lessons

   The short handshake timeout had been chosen to detect replica failures quickly, and AWS concluded it was too low. Nodes removing themselves from the cluster right after several handshake timeouts widened the event and caused the API errors; the nodes should have waited and retried with a longer timeout.

   View case

By root cause2