Cases2 cases

1 to 2 of 2

1. IL-0499 Discord voice and video calls hung at Awaiting Endpoint for three hours after 17% of sessions stopped at once

   Discord2026

   Symptoms

   From 12:13 to 15:30 PDT on 25 March 2026, voice and video on Discord were heavily degraded. Users mostly could not start or join calls and saw "Awaiting Endpoint" in their call status.

   Service outage

   Trigger

   A routine change to tune the sessions service cluster on Kubernetes gave each pod more CPU and memory and lowered the pod count in proportion. When it was deployed to the first zone at 12:13 PDT, Kubernetes terminated 50% of that zone's pods, and the termination grace period ran out before the sessions could be handed off.

   Root cause

   Configuration change

   Component

   Pod terminationCoordination and key-value storeDatabase proxy and connection pooler

   Contributing factor

   Retry stormUnknown limit

   Remediation

   • Set low syncer spawn rate limits in the calls, streams and guilds services and restarted the voice syncers cluster
   • Restarted voice syncers instances one at a time, and four of the first five recovered fully
   • Brought 15 new voice syncers instances online, doubling capacity and halving the syncers per instance
   • Restarted the last backed-up voice syncers instance, leaving the doubled cluster fully healthy

   Action items

   • Add a validating admission webhook that rejects a scale-down of Elixir workloads until each pod has handed off its processes
   • Replace the Holster.Pool supervisor with a PartitionSupervisor so outgoing connections are created and checked out concurrently
   • Move gun connection process lifecycle management into Holster.Pool so the gun connection supervisor is no longer used
   • Tune the upstream rate limits that call into voice syncers, with load shedding tooling that can force them
   • Add rate limits on endpoint selection in voice syncers to limit outbound connection creation
   • Monitor the HTTP connection pooling mailboxes
   • Review and expand instrumentation and logging for service discovery announcement and voice syncer RPCs to SFUs
   • Monitor the most critical processes across all services instead of aggregate metrics such as overall mailbox health
   • Bring more workloads into Kubernetes so capacity can be scaled with a single command instead of provisioning VMs by hand
   • Re-architect the A/V system to scale its control plane services horizontally across multiple regions

   Lessons

   Discord says a large enough traffic spike will find a bottleneck somewhere; in earlier large-load incidents its ScyllaDB databases limited recovery, and this time the bottleneck moved to the voice services. Facing a capacity constraint, the team asks whether to increase supply, such as adding services, or reduce demand, such as rate limiting inbound traffic. During an incident it matters to know the toolbox, and Elixir and the BEAM give engineers many introspection options. Any slowdown in the hot loop of a single-threaded process can degrade both the service and the user experience.

   View case
2. IL-0271 An empty etcd service entry crashed Discord's Elixir chat nodes, and Discord was down for most users for an hour

   Discord2020sev1

   Symptoms

   On 16 March 2020 at 14:19 PDT most of Discord's Elixir services, which handle real-time connections and chat message processing, crashed at once. Discord lost much of its real-time capacity, the failure cascaded into dependent systems, and most users could not connect for about an hour.

   Service outageHigh error rate

   Trigger

   At 14:18 the Google Cloud managed instance group autoscaler added a node to Discord's API cluster, and the TCP connection of its first announce request to etcd was reset after the request headers were sent but before the body.

   Root cause

   Software defect

   Component

   Coordination and key-value storeService discovery

   Contributing factor

   Automation amplification

   Detection

   At 14:21 several internal alarms fired on a drop in key metrics, along with anomaly alerts for cluster utilization, and a SEV1 incident was declared.

   Remediation

   • Restarted the lost real-time capacity and repaired the failed nodes
   • Throttled connections to Discord to help services recover
   • Stopped anti-entropy repairs on two ScyllaDB clusters so reconnecting users would not starve them of resources
   • Raised the connection throttle step by step to let more users reconnect

   Action items

   • Make the etcd clients send each announce request in a single TCP packet, so headers and body arrive together or not at all
   • Add error handling that ignores services whose key value is corrupt instead of crash-looping
   • File an upstream bug report with the Go project about net/http not checking the body against Content-Length

   Lessons

   Discord's service discovery code handled many etcd failure modes but did not expect a client to write a key without a value because of a transient network error. Go's net/http accepts a request whose body is shorter than its Content-Length header, which breaks the HTTP 1.1 specification and added a failure case nobody had anticipated.

   View case

By root cause2