Incident Library
Applied filters Root causeConfiguration changeComponentCoordination and key-value storeFactorUnknown limitClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0499 Discord voice and video calls hung at Awaiting Endpoint for three hours after 17% of sessions stopped at once
    Discord2026
    Symptoms

    From 12:13 to 15:30 PDT on 25 March 2026, voice and video on Discord were heavily degraded. Users mostly could not start or join calls and saw "Awaiting Endpoint" in their call status.

    Service outage
    Trigger

    A routine change to tune the sessions service cluster on Kubernetes gave each pod more CPU and memory and lowered the pod count in proportion. When it was deployed to the first zone at 12:13 PDT, Kubernetes terminated 50% of that zone's pods, and the termination grace period ran out before the sessions could be handed off.

    Root cause
    Configuration change
    Component
    Pod terminationCoordination and key-value storeDatabase proxy and connection pooler
    Contributing factor
    Retry stormUnknown limit
    Remediation
    • Set low syncer spawn rate limits in the calls, streams and guilds services and restarted the voice syncers cluster
    • Restarted voice syncers instances one at a time, and four of the first five recovered fully
    • Brought 15 new voice syncers instances online, doubling capacity and halving the syncers per instance
    • Restarted the last backed-up voice syncers instance, leaving the doubled cluster fully healthy
    Action items
    • Add a validating admission webhook that rejects a scale-down of Elixir workloads until each pod has handed off its processes
    • Replace the Holster.Pool supervisor with a PartitionSupervisor so outgoing connections are created and checked out concurrently
    • Move gun connection process lifecycle management into Holster.Pool so the gun connection supervisor is no longer used
    • Tune the upstream rate limits that call into voice syncers, with load shedding tooling that can force them
    • Add rate limits on endpoint selection in voice syncers to limit outbound connection creation
    • Monitor the HTTP connection pooling mailboxes
    • Review and expand instrumentation and logging for service discovery announcement and voice syncer RPCs to SFUs
    • Monitor the most critical processes across all services instead of aggregate metrics such as overall mailbox health
    • Bring more workloads into Kubernetes so capacity can be scaled with a single command instead of provisioning VMs by hand
    • Re-architect the A/V system to scale its control plane services horizontally across multiple regions
    Lessons

    Discord says a large enough traffic spike will find a bottleneck somewhere; in earlier large-load incidents its ScyllaDB databases limited recovery, and this time the bottleneck moved to the voice services. Facing a capacity constraint, the team asks whether to increase supply, such as adding services, or reduce demand, such as rate limiting inbound traffic. During an incident it matters to know the toolbox, and Elixir and the BEAM give engineers many introspection options. Any slowdown in the hot loop of a single-threaded process can degrade both the service and the user experience.

    View case
Check your cluster for this

This case involves Coordination and key-value store and Database proxy and connection pooler. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Configuration change1 case

Component
Coordination and key-value storeDatabase proxy and connection poolerPod termination
Contributing factor
Unknown limitRetry storm
Organisation
Discord
  1. IL-0499 Discord voice and video calls hung at Awaiting Endpoint for three hours after 17% of sessions stopped at once
    Discord2026