Cases1 case
1 to 1 of 1
- Symptoms
On 16 March 2020 at 14:19 PDT most of Discord's Elixir services, which handle real-time connections and chat message processing, crashed at once. Discord lost much of its real-time capacity, the failure cascaded into dependent systems, and most users could not connect for about an hour.
- Trigger
At 14:18 the Google Cloud managed instance group autoscaler added a node to Discord's API cluster, and the TCP connection of its first announce request to etcd was reset after the request headers were sent but before the body.
- Root cause
- Component
- Contributing factor
- Detection
At 14:21 several internal alarms fired on a drop in key metrics, along with anomaly alerts for cluster utilization, and a SEV1 incident was declared.
- Remediation
- Restarted the lost real-time capacity and repaired the failed nodes
- Throttled connections to Discord to help services recover
- Stopped anti-entropy repairs on two ScyllaDB clusters so reconnecting users would not starve them of resources
- Raised the connection throttle step by step to let more users reconnect
- Action items
- Make the etcd clients send each announce request in a single TCP packet, so headers and body arrive together or not at all
- Add error handling that ignores services whose key value is corrupt instead of crash-looping
- File an upstream bug report with the Go project about net/http not checking the body against Content-Length
- Lessons
Discord's service discovery code handled many etcd failure modes but did not expect a client to write a key without a value because of a transient network error. Go's net/http accepts a request whose body is shorter than its Content-Length header, which breaks the HTTP 1.1 specification and added a failure case nobody had anticipated.
This case involves Coordination and key-value store and Service discovery. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.