Incident Library
Applied filters Root causeConfiguration changeComponentCoordination and key-value storeClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0499 Discord voice and video calls hung at Awaiting Endpoint for three hours after 17% of sessions stopped at once
    Discord2026
    Symptoms

    From 12:13 to 15:30 PDT on 25 March 2026, voice and video on Discord were heavily degraded. Users mostly could not start or join calls and saw "Awaiting Endpoint" in their call status.

    Service outage
    Trigger

    A routine change to tune the sessions service cluster on Kubernetes gave each pod more CPU and memory and lowered the pod count in proportion. When it was deployed to the first zone at 12:13 PDT, Kubernetes terminated 50% of that zone's pods, and the termination grace period ran out before the sessions could be handed off.

    Root cause
    Configuration change
    Component
    Pod terminationCoordination and key-value storeDatabase proxy and connection pooler
    Contributing factor
    Retry stormUnknown limit
    Remediation
    • Set low syncer spawn rate limits in the calls, streams and guilds services and restarted the voice syncers cluster
    • Restarted voice syncers instances one at a time, and four of the first five recovered fully
    • Brought 15 new voice syncers instances online, doubling capacity and halving the syncers per instance
    • Restarted the last backed-up voice syncers instance, leaving the doubled cluster fully healthy
    Action items
    • Add a validating admission webhook that rejects a scale-down of Elixir workloads until each pod has handed off its processes
    • Replace the Holster.Pool supervisor with a PartitionSupervisor so outgoing connections are created and checked out concurrently
    • Move gun connection process lifecycle management into Holster.Pool so the gun connection supervisor is no longer used
    • Tune the upstream rate limits that call into voice syncers, with load shedding tooling that can force them
    • Add rate limits on endpoint selection in voice syncers to limit outbound connection creation
    • Monitor the HTTP connection pooling mailboxes
    • Review and expand instrumentation and logging for service discovery announcement and voice syncer RPCs to SFUs
    • Monitor the most critical processes across all services instead of aggregate metrics such as overall mailbox health
    • Bring more workloads into Kubernetes so capacity can be scaled with a single command instead of provisioning VMs by hand
    • Re-architect the A/V system to scale its control plane services horizontally across multiple regions
    Lessons

    Discord says a large enough traffic spike will find a bottleneck somewhere; in earlier large-load incidents its ScyllaDB databases limited recovery, and this time the bottleneck moved to the voice services. Facing a capacity constraint, the team asks whether to increase supply, such as adding services, or reduce demand, such as rate limiting inbound traffic. During an incident it matters to know the toolbox, and Elixir and the BEAM give engineers many introspection options. Any slowdown in the hot loop of a single-threaded process can degrade both the service and the user experience.

    View case
  2. IL-0374 DigitalOcean creates and management actions failed globally when its service discovery datastore lost quorum
    DigitalOcean2024
    Symptoms

    From 16:30 UTC on 5 August 2024, DigitalOcean's internal services could not discover the other services they depended on. Creates, event processing and the management of DigitalOcean products failed in all regions through the Cloud Control Panel and the API, while existing customer resources such as Droplets stayed online.

    Service outageCluster or node provisioning stall
    Trigger

    During the rollout of a new datastore cluster in a new datacenter, a replication configuration was deployed against the datastore behind internal service discovery that gave the new datacenter, which held zero keys, 100% ownership of all keys.

    Root cause
    Configuration change
    Component
    Coordination and key-value storeService discoveryDatabase replication
    Contributing factor
    Excessive blast radius
    Detection

    Service discovery unavailability was first reported internally at 16:35 UTC, five minutes after the rollout of the new datastore cluster began.

    Remediation
    • Reverted the replication configuration and ran it against the region that had become the datastore owner
    • Applied the configuration by hand on the nodes of lagging regions and ran local data repairs
    • Cleaned up failed events and processed pending events that had not timed out
    Action items
    • Perform this kind of datastore extension inside a declared maintenance window, announced on the status page
    • Update the process documentation for this type of deployment, stage by stage
    • Automate the manual steps used in this deployment to reduce human error
    • Evaluate the internal datastore topology and whether to regionalise or layer it
    View case
Check your cluster for this

These 2 cases involve Coordination and key-value store and Database proxy and connection pooler. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Configuration change2 cases

Component
Coordination and key-value store2Database proxy and connection pooler1Database replication1Pod termination1Service discovery1
Contributing factor
Excessive blast radius1Unknown limit1Retry storm1
Organisation
DigitalOcean1Discord1
  1. IL-0499 Discord voice and video calls hung at Awaiting Endpoint for three hours after 17% of sessions stopped at once
    Discord2026
  2. IL-0374 DigitalOcean creates and management actions failed globally when its service discovery datastore lost quorum
    DigitalOcean2024