IL-0499 Discord · 2026-03-25kubernetesgcp

Discord voice and video calls hung at Awaiting Endpoint for three hours after 17% of sessions stopped at once

Discord's Realtime Infrastructure team was moving its stateful Elixir services to Kubernetes and was finishing the migration of the sessions service, which runs one session process for every connected device. Before a host goes offline, its processes must hand their data to another node, and deployments wait until an entity count on the server reaches zero. To lower CPU usage that ran high on weekends, the team deployed a change that gave the session pods more CPU and memory and reduced the pod count in proportion. In the first zone, us-east1-b, Kubernetes terminated 50% of the pods. The service tries to hand off its processes when Kubernetes signals it, but a safety check that waits for in-progress events held the handoff until the termination grace period had passed. Because the service runs in three equally balanced zones, 17% of all sessions stopped without a handoff.

Process monitors sent a :DOWN message for every stopped session to each process watching it. The gateway service told the affected clients to reconnect, and the start session rate limit, not yet re-tuned after the migration, let more concurrent session starts through because there were more pods than in the old VM setup; the gateway nodes in us-east1-b ran out of memory. As sessions left and rejoined calls, the voice syncers service, which decides where calls run and sends RPC commands to more than 25,000 SFU instances, had to open a flood of outbound HTTPS connections. These go through Holster, an internal library that pools connections from the Erlang HTTP client gun, and every new connection passed through two single-process Erlang supervisors, one for Holster.Pool and one for all gun connections. Each spawn does a selective receive over the supervisor's mailbox, which adds about 1ms with a mailbox queue of about 100k, so the backlog could not be cleared. The etcd connection used for service discovery went through the same supervisors, so voice syncers instances could not refresh their registration and dropped out of etcd when the 60s time-to-live expired.

Targeted restarts and a full cluster restart recovered the instances only briefly. The call owner services' existing limits on voice syncer creation turned out to be outdated or unset, and by 13:43 PDT the calls, streams and guilds services had low syncer spawn limits and the cluster was restarted again. This time the gun supervisor backed up instead of Holster.Pool, the instances stayed announced in etcd and part of the RPCs to SFUs succeeded. Targeted restarts from 14:03 then fully recovered instances, and at 14:15 15 new voice syncers instances, provisioned on GCP VMs through Terraform and Salt and added by hand as ring candidates in etcd, came online and doubled the cluster. The last backed-up instance was restarted at 14:26 and the doubled cluster was healthy.

Severity: –
Scope: Voice and video calls on Discord: users mostly could not start or join calls; 17% of sessions stopped and gateway connections in us-east1-b were cut
Users affected: –
Region: us-east1-b for sessions and the gateway; voice syncer traffic globally
Time to detect: –
Time to resolve: 3h 17m

Symptoms

From 12:13 to 15:30 PDT on 25 March 2026, voice and video on Discord were heavily degraded. Users mostly could not start or join calls and saw "Awaiting Endpoint" in their call status.

It started when a deploy of the sessions service on Kubernetes stopped 17% of all Discord sessions without handing them off. The gateway service, which holds the websocket connections, told the affected clients to reconnect, and the flood of session starts made memory spike on all gateway nodes in us-east1-b until they ran out of memory and restarted. That cut the gateway connections of the remaining users in us-east1-b, who reconnected again through us-east1-c and us-east1-d.

From 12:13 PDT the aggregate process mailbox length on all 15 voice syncers instances grew and RPCs from voice syncers to the SFUs fell sharply. The instances dropped out of the etcd-based consistent hash ring, stopped taking on new syncers and could not make RPC requests. A snapshot of the running Erlang processes taken with Recon, sorted by mailbox queue length, put the Holster pool's DynamicSupervisor first.

The first recovery attempts did not last. Restarting the voice syncers application on instance 2-13 at 12:43 PDT and restarting the Holster.Pool DynamicSupervisor on 2-1 at 12:47 brought each instance back only briefly, because the few healthy instances also took the traffic of the unhealthy ones as secondary and tertiary placements and their mailboxes grew again. A restart of the whole voice syncers cluster at 13:05 recreated millions of active calls at once, and by 13:09 all restarted nodes had growing mailboxes again.

Symptom Service outage

Root cause

Root cause Configuration change

Component Pod termination Coordination and key-value store Database proxy and connection pooler

Contributing factor Retry storm Unknown limit

Trigger: A routine change to tune the sessions service cluster on Kubernetes gave each pod more CPU and memory and lowered the pod count in proportion. When it was deployed to the first zone at 12:13 PDT, Kubernetes terminated 50% of that zone's pods, and the termination grace period ran out before the sessions could be handed off.

Detection

–

Remediation

Set low syncer spawn rate limits in the calls, streams and guilds services and restarted the voice syncers cluster
Restarted voice syncers instances one at a time, and four of the first five recovered fully
Brought 15 new voice syncers instances online, doubling capacity and halving the syncers per instance
Restarted the last backed-up voice syncers instance, leaving the doubled cluster fully healthy

Action items

Add a validating admission webhook that rejects a scale-down of Elixir workloads until each pod has handed off its processes
Replace the Holster.Pool supervisor with a PartitionSupervisor so outgoing connections are created and checked out concurrently
Move gun connection process lifecycle management into Holster.Pool so the gun connection supervisor is no longer used
Tune the upstream rate limits that call into voice syncers, with load shedding tooling that can force them
Add rate limits on endpoint selection in voice syncers to limit outbound connection creation
Monitor the HTTP connection pooling mailboxes
Review and expand instrumentation and logging for service discovery announcement and voice syncer RPCs to SFUs
Monitor the most critical processes across all services instead of aggregate metrics such as overall mailbox health
Bring more workloads into Kubernetes so capacity can be scaled with a single command instead of provisioning VMs by hand
Re-architect the A/V system to scale its control plane services horizontally across multiple regions

Lessons

Discord says a large enough traffic spike will find a bottleneck somewhere; in earlier large-load incidents its ScyllaDB databases limited recovery, and this time the bottleneck moved to the voice services. Facing a capacity constraint, the team asks whether to increase supply, such as adding services, or reduce demand, such as rate limiting inbound traffic. During an incident it matters to know the toolbox, and Elixir and the BEAM give engineers many introspection options. Any slowdown in the hot loop of a single-threaded process can degrade both the service and the user experience.

Sources

https://discord.com/blog/behind-the-scenes-of-the-3-25-26-voice-outage · blog · archived copy

Related cases

IL-0574 Aurora MySQL writer CPU stuck near 100% minutes after Safie pointed production at a new Amazon RDS ProxySafie · 2026
same root cause · shared Database proxy and connection pooler
IL-0374 DigitalOcean creates and management actions failed globally when its service discovery datastore lost quorumDigitalOcean · 2024
same root cause · shared Coordination and key-value store

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Pod termination, Coordination and key-value store, Database proxy and connection pooler among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0499
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0499/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.