Incident Library

IL-0271 Discord · 2020-03-16databasegcp

An empty etcd service entry crashed Discord's Elixir chat nodes, and Discord was down for most users for an hour

Discord stores service discovery information in etcd through the etcd v2 API. A discoverable service announces itself by setting a key with a 60-second TTL in its cluster's directory and keeps re-announcing it as a heartbeat; the key's value is JSON metadata. Discord's real-time chat services use service discovery to find the RPC endpoint they call to load data from the databases when a user connects or when a guild is created or reloaded. The etcd client uses Python's built-in HTTP client, which sends the PUT request with its Content-Length header in one packet and the urlencoded body in a second packet. etcd's v2 keys API parses the form with the Go standard library's net/http Request.ParseForm() method, which does not check that the body length matches the Content-Length header.

At 14:18 PDT a new API node, api-prd-main-m1ds, joined the API cluster during normal autoscaling before peak hours. Its connection was reset between the two packets, so etcd stored the key with an empty string as the value. The Elixir nodes watching the "discord_api" service failed to deserialize the entry and crash-looped, and the supervisor's "max restart intensity" restarted whole nodes on about half of the watching nodes across several clusters. The crashes lasted until the API node retried its announcement and rewrote the key, but by then Discord had lost most of its real-time capacity and the failure had cascaded into dependent systems. A SEV1 incident was declared at 14:21.

Engineers restarted the lost capacity, throttled connections to Discord and stopped anti-entropy repairs on two ScyllaDB clusters so the reconnecting users would not starve them of resources. From 15:07 to 15:26 the connection throttle was raised step by step as services recovered, and at 15:32 the service was declared fully operational. A mitigation was deployed at 23:00, and engineers reproduced the failure in the development environment.

Severity
sev1
Scope
Discord real-time connections and chat; Discord was unavailable for most users for about an hour and the API returned increased errors
Users affected
most Discord users
Region
–
Time to detect
3m
Time to resolve
1h 14m

Symptoms

On 16 March 2020 at 14:19 PDT most of Discord's Elixir services, which handle real-time connections and chat message processing, crashed at once. Discord lost much of its real-time capacity, the failure cascaded into dependent systems, and most users could not connect for about an hour.

At 14:21 several internal alarms fired on a drop in key metrics, together with anomaly alerts for cluster utilization. The status page reported that a number of Discord servers were completely unavailable, and later that a series of fatal errors had made the majority of servers unavailable. A minute before the crash, a newly added API node, api-prd-main-m1ds, had logged "http.client.RemoteDisconnected: Remote end closed connection without response" while announcing itself to service discovery. Almost all Elixir nodes then logged that the service watcher for the "discord_api" service had crashed while parsing the JSON metadata stored in the key's value in etcd.

The service discovery processes crash-looped on the invalid data. Their supervisor saw them crashing too often, hit "max restart intensity" and restarted the whole node, which happened at the same moment on about 50% of the nodes watching for API nodes, across several clusters. By 14:55 engineers had tied the outage to a spike of errors from the service discovery modules. The first explanation was that Discord had hit a cap on the number of watchers in etcd, and engineers tried to raise it through runtime configuration while they kept repairing failed nodes.

Symptom Service outageHigh error rate

Root cause

Root cause Software defect

Component Coordination and key-value storeService discovery
Contributing factor Automation amplification
Trigger
At 14:18 the Google Cloud managed instance group autoscaler added a node to Discord's API cluster, and the TCP connection of its first announce request to etcd was reset after the request headers were sent but before the body.

Detection

At 14:21 several internal alarms fired on a drop in key metrics, along with anomaly alerts for cluster utilization, and a SEV1 incident was declared.

Remediation

Action items

Lessons

Discord's service discovery code handled many etcd failure modes but did not expect a client to write a key without a value because of a transient network error. Go's net/http accepts a request whose body is shorter than its Content-Length header, which breaks the HTTP 1.1 specification and added a failure case nobody had anticipated.

Sources

References

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Coordination and key-value store, Service discovery among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.