Incident Library
Applied filters SymptomHigh error rateFactorAutomation amplificationOrganisationDiscordClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0271 An empty etcd service entry crashed Discord's Elixir chat nodes, and Discord was down for most users for an hour
    Discord2020sev1
    Symptoms

    On 16 March 2020 at 14:19 PDT most of Discord's Elixir services, which handle real-time connections and chat message processing, crashed at once. Discord lost much of its real-time capacity, the failure cascaded into dependent systems, and most users could not connect for about an hour.

    Service outageHigh error rate
    Trigger

    At 14:18 the Google Cloud managed instance group autoscaler added a node to Discord's API cluster, and the TCP connection of its first announce request to etcd was reset after the request headers were sent but before the body.

    Root cause
    Software defect
    Component
    Coordination and key-value storeService discovery
    Contributing factor
    Automation amplification
    Detection

    At 14:21 several internal alarms fired on a drop in key metrics, along with anomaly alerts for cluster utilization, and a SEV1 incident was declared.

    Remediation
    • Restarted the lost real-time capacity and repaired the failed nodes
    • Throttled connections to Discord to help services recover
    • Stopped anti-entropy repairs on two ScyllaDB clusters so reconnecting users would not starve them of resources
    • Raised the connection throttle step by step to let more users reconnect
    Action items
    • Make the etcd clients send each announce request in a single TCP packet, so headers and body arrive together or not at all
    • Add error handling that ignores services whose key value is corrupt instead of crash-looping
    • File an upstream bug report with the Go project about net/http not checking the body against Content-Length
    Lessons

    Discord's service discovery code handled many etcd failure modes but did not expect a client to write a key without a value because of a transient network error. Go's net/http accepts a request whose body is shorter than its Content-Length header, which breaks the HTTP 1.1 specification and added a failure case nobody had anticipated.

    View case
Check your cluster for this

This case involves Coordination and key-value store and Service discovery. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Software defect1 case

Component
Coordination and key-value storeService discovery
Contributing factor
Automation amplification
Organisation
Discord
  1. IL-0271 An empty etcd service entry crashed Discord's Elixir chat nodes, and Discord was down for most users for an hour
    Discord2020