Graph
Cases3 cases
1 to 3 of 3
IL-0499Discord voice and video calls hung at Awaiting Endpoint for three hours after 17% of sessions stopped at onceDiscord2026- Symptoms
From 12:13 to 15:30 PDT on 25 March 2026, voice and video on Discord were heavily degraded. Users mostly could not start or join calls and saw "Awaiting Endpoint" in their call status.
- Trigger
A routine change to tune the sessions service cluster on Kubernetes gave each pod more CPU and memory and lowered the pod count in proportion. When it was deployed to the first zone at 12:13 PDT, Kubernetes terminated 50% of that zone's pods, and the termination grace period ran out before the sessions could be handed off.
- Root cause
- Component
- Contributing factor
- Remediation
- Set low syncer spawn rate limits in the calls, streams and guilds services and restarted the voice syncers cluster
- Restarted voice syncers instances one at a time, and four of the first five recovered fully
- Brought 15 new voice syncers instances online, doubling capacity and halving the syncers per instance
- Restarted the last backed-up voice syncers instance, leaving the doubled cluster fully healthy
- Action items
- Add a validating admission webhook that rejects a scale-down of Elixir workloads until each pod has handed off its processes
- Replace the Holster.Pool supervisor with a PartitionSupervisor so outgoing connections are created and checked out concurrently
- Move gun connection process lifecycle management into Holster.Pool so the gun connection supervisor is no longer used
- Tune the upstream rate limits that call into voice syncers, with load shedding tooling that can force them
- Add rate limits on endpoint selection in voice syncers to limit outbound connection creation
- Monitor the HTTP connection pooling mailboxes
- Review and expand instrumentation and logging for service discovery announcement and voice syncer RPCs to SFUs
- Monitor the most critical processes across all services instead of aggregate metrics such as overall mailbox health
- Bring more workloads into Kubernetes so capacity can be scaled with a single command instead of provisioning VMs by hand
- Re-architect the A/V system to scale its control plane services horizontally across multiple regions
- Lessons
Discord says a large enough traffic spike will find a bottleneck somewhere; in earlier large-load incidents its ScyllaDB databases limited recovery, and this time the bottleneck moved to the voice services. Facing a capacity constraint, the team asks whether to increase supply, such as adding services, or reduce demand, such as rate limiting inbound traffic. During an incident it matters to know the toolbox, and Elixir and the BEAM give engineers many introspection options. Any slowdown in the hot loop of a single-threaded process can degrade both the service and the user experience.
IL-0277Patching Elastic Cloud's ZooKeeper ensemble in us-east-1 lost quorum, and hosted clusters and Kibana went unreachableElastic2019- Symptoms
On 4 February 2019 at about 02:50 UTC, Elastic Cloud customers with deployments in AWS us-east-1 lost stable access to their Elasticsearch clusters, and an increase in HTTP 5xx errors showed on the load balancer. Deployments were partially or completely unavailable until 09:28 UTC, Kibana access was disrupted for most customers in the region, and the Elastic Cloud User Console timed out when customers tried to view their deployments.
- Trigger
A routine patching procedure in AWS us-east-1 replaced the first and second members of the three-node ZooKeeper ensemble with patched hosts between 02:25 and 02:45 UTC, following the same documented process used in the other 12 regions.
- Root cause
- Component
- Contributing factor
- Detection
An increase in HTTP 5xx errors on the load balancer was identified at 02:50 UTC, the time the ZooKeeper ensemble lost its leader.
- Remediation
- Restarted internal services in sweeping restarts at 03:22 and 06:00 to reduce client connection problems
- Isolated the ZooKeeper ensemble by cutting off client connections
- Rolled back the remaining ensemble member after its host became unstable
- Rolled back the second ensemble member after its host became unstable
- Rebooted the ZooKeeper hosts made unresponsive by the runc CPU softlocks
- Re-established ZooKeeper quorum by removing client load and operator debugging, then reintroduced client load gradually
- Increased the initLimit setting of the ZooKeeper observer layer used by the proxies
- Scaled the proxy fleet up and out to handle the increased traffic
- Applied a runbook that restarted the internal proxying containers and Kibana instances and set sysctl limits
- Action items
- Reduce the dataset size carried by the ZooKeeper ensembles by over 60%
- Reduce the number of health checks and intervals needed before a healthy proxy node joins the load balancing layer
- Stop counting ZooKeeper connectivity in the proxy health checks once a proxy has its configuration and routes
- Canary the rewritten proxy v2 architecture, which decouples the proxy service from ZooKeeper state, in production regions
- Improve visibility into ZooKeeper with better logging and metrics for the ZooKeeper processes, the JVM and the container and host systems
- Make Kibana route discovery more resilient
- Fix the Kibana issues that amplify internal HTTP requests during availability events
- Fix the Kibana issues that leak connections
- Make internal monitoring agents use backoff with jitter
- Formalize a maintenance matrix with set timeframes per region, so failures of critical services fall in non-critical times
- Give runbooks for coordination layer maintenance a better decision criteria matrix
- Set up a feedback loop with support leadership for incident communications through support and the status page
- Lessons
Service metrics that reported the ZooKeeper hosts healthy were not enough to show the state of individual hosts or of the ensemble, so they gave a false signal that maintenance could continue. Because the platform depended on the coordination layer, losing its quorum caused a major service disruption in which every Elasticsearch cluster in the region saw some failures, and fixes that worked one by one fell short at the scale of the failure.
IL-0587Elastic Cloud proxies in AWS eu-west-1 flooded an overloaded ZooKeeper and ran out of memory, cutting cluster accessElastic2019- Symptoms
On 18 January 2019 from 09:09 UTC the leader of the three-node Apache ZooKeeper ensemble behind Elastic Cloud in AWS eu-west-1 (Ireland) ran at heavy CPU load, and its clients lost their connections. From about 10:08 UTC customers in the region had severely degraded access to their Elasticsearch and Kibana deployments for roughly 3 hours, and for about 20 minutes every deployment in the region was unavailable.
- Trigger
At 09:09 UTC the leader server of the ZooKeeper ensemble in the eu-west-1 coordination layer hit a high CPU threshold, which caused client disconnections.
- Root cause
- Component
- Contributing factor
- Detection
The on-call SREs received alerts about service degradation in the coordination layer at 09:35 UTC, after the high CPU on the ZooKeeper leader had caused client disconnections, and started a production incident.
- Remediation
- Scaled the proxy layer up and out to take the surging client traffic hitting the load balancers
- Triggered a leader election in the ZooKeeper ensemble and stabilised it
- Action items
- Replace Curator's TreeCache with TreeWatcher across all Elasticsearch Service providers and regions
- Contribute TreeWatcher upstream to the Apache Curator project
- Roll out a new proxy layer that keeps the routing table in a versioned local file, so it can route traffic while the ZooKeeper connection is lost or flaky
These 3 cases involve Coordination and key-value store and conntrack table. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.
By root cause2
Configuration change1 case
- Contributing factor
- Organisation
Software defect2 cases
- Component
- Contributing factor
- Organisation