Cases9 cases

1 to 5 of 9

1. IL-0499 Discord voice and video calls hung at Awaiting Endpoint for three hours after 17% of sessions stopped at once

   Discord2026

   Symptoms

   From 12:13 to 15:30 PDT on 25 March 2026, voice and video on Discord were heavily degraded. Users mostly could not start or join calls and saw "Awaiting Endpoint" in their call status.

   Service outage

   Trigger

   A routine change to tune the sessions service cluster on Kubernetes gave each pod more CPU and memory and lowered the pod count in proportion. When it was deployed to the first zone at 12:13 PDT, Kubernetes terminated 50% of that zone's pods, and the termination grace period ran out before the sessions could be handed off.

   Root cause

   Configuration change

   Component

   Pod terminationCoordination and key-value storeDatabase proxy and connection pooler

   Contributing factor

   Retry stormUnknown limit

   Remediation

   • Set low syncer spawn rate limits in the calls, streams and guilds services and restarted the voice syncers cluster
   • Restarted voice syncers instances one at a time, and four of the first five recovered fully
   • Brought 15 new voice syncers instances online, doubling capacity and halving the syncers per instance
   • Restarted the last backed-up voice syncers instance, leaving the doubled cluster fully healthy

   Action items

   • Add a validating admission webhook that rejects a scale-down of Elixir workloads until each pod has handed off its processes
   • Replace the Holster.Pool supervisor with a PartitionSupervisor so outgoing connections are created and checked out concurrently
   • Move gun connection process lifecycle management into Holster.Pool so the gun connection supervisor is no longer used
   • Tune the upstream rate limits that call into voice syncers, with load shedding tooling that can force them
   • Add rate limits on endpoint selection in voice syncers to limit outbound connection creation
   • Monitor the HTTP connection pooling mailboxes
   • Review and expand instrumentation and logging for service discovery announcement and voice syncer RPCs to SFUs
   • Monitor the most critical processes across all services instead of aggregate metrics such as overall mailbox health
   • Bring more workloads into Kubernetes so capacity can be scaled with a single command instead of provisioning VMs by hand
   • Re-architect the A/V system to scale its control plane services horizontally across multiple regions

   Lessons

   Discord says a large enough traffic spike will find a bottleneck somewhere; in earlier large-load incidents its ScyllaDB databases limited recovery, and this time the bottleneck moved to the voice services. Facing a capacity constraint, the team asks whether to increase supply, such as adding services, or reduce demand, such as rate limiting inbound traffic. During an incident it matters to know the toolbox, and Elixir and the BEAM give engineers many introspection options. Any slowdown in the hot loop of a single-threaded process can degrade both the service and the user experience.

   View case
2. IL-0374 DigitalOcean creates and management actions failed globally when its service discovery datastore lost quorum

   DigitalOcean2024

   Symptoms

   From 16:30 UTC on 5 August 2024, DigitalOcean's internal services could not discover the other services they depended on. Creates, event processing and the management of DigitalOcean products failed in all regions through the Cloud Control Panel and the API, while existing customer resources such as Droplets stayed online.

   Service outageCluster or node provisioning stall

   Trigger

   During the rollout of a new datastore cluster in a new datacenter, a replication configuration was deployed against the datastore behind internal service discovery that gave the new datacenter, which held zero keys, 100% ownership of all keys.

   Root cause

   Configuration change

   Component

   Coordination and key-value storeService discoveryDatabase replication

   Contributing factor

   Excessive blast radius

   Detection

   Service discovery unavailability was first reported internally at 16:35 UTC, five minutes after the rollout of the new datastore cluster began.

   Remediation

   • Reverted the replication configuration and ran it against the region that had become the datastore owner
   • Applied the configuration by hand on the nodes of lagging regions and ran local data repairs
   • Cleaned up failed events and processed pending events that had not timed out

   Action items

   • Perform this kind of datastore extension inside a declared maintenance window, announced on the status page
   • Update the process documentation for this type of deployment, stage by stage
   • Automate the manual steps used in this deployment to reduce human error
   • Evaluate the internal datastore topology and whether to regionalise or layer it

   View case
3. IL-0514 PyPI's backends stopped responding for 7 minutes as rescheduled pods could not start while Consul and Vault were down

   Python Software Foundation2022

   Symptoms

   At 18:23 UTC on 28 July 2022, PSF Infrastructure got an alert that at least one Vault container in PyPI's Kubernetes cluster was down and needed unsealing. At 18:46 UTC PyPI's backends became unresponsive, and they stayed down until 18:53 UTC.

   Service outagePods stuck in Terminating

   Trigger

   From 17:24 UTC the cloud provider automatically replaced several instances of PyPI's Kubernetes cluster because of an unknown failure. All replacements were healthy by 18:20 UTC, but the event reshuffled many pods and restarted Consul and Vault on new nodes.

   Root cause

   Cloud provider outage

   Component

   Coordination and key-value storeSecrets store

   Contributing factor

   Automation amplification

   Detection

   An alert at 18:23 UTC told PSF Infrastructure that at least one Vault container was down and needed unsealing.

   Remediation

   • Forcefully removed the Consul pod stuck in Terminating so it was recreated and Consul could elect a leader
   • Unsealed the new Vault containers

   View case
4. IL-0213 Roblox was down for 73 hours because key-value writes on its single Consul cluster slowed from under 300 ms to 2 seconds

   Roblox2021

   Symptoms

   On the afternoon of 28 October 2021 (13:37 PST), Vault performance at Roblox degraded and a single Consul server showed high CPU load. Consul cluster metrics showed that writes to its key-value (KV) store had a 50th percentile latency of 2 seconds instead of the usual under 300 ms. Players were not yet affected.

   High latencySustained high node CPUService outageMonitoring outage

   Trigger

   On 27 October at 14:00, one day before the outage, Roblox enabled the Consul streaming feature on the backend service responsible for traffic routing and, to prepare for end-of-year traffic, added 50% more nodes to traffic routing.

   Root cause

   Software defect

   Component

   Coordination and key-value storeService discoveryIn-memory cache

   Contributing factor

   Single point of failureOverlapping failuresMonitoring blind spotHarmful manual actionChange with a delayed effect

   Detection

   Roblox engineers began investigating when Vault performance degraded and a single Consul server had high CPU load, before players were affected. Consul cluster metrics then showed elevated KV write latency.

   Remediation

   • Replaced one Consul cluster node on the suspicion of degraded hardware
   • Replaced all Consul nodes with 128-core machines with faster NVMe SSDs
   • Shut down the Consul cluster and reset its state from a snapshot taken at the start of the outage, with access blocked by iptables
   • Blocked external traffic, disabled all non-essential Consul use, scaled services down and lowered health check frequency from 60 seconds to 10 minutes
   • Moved the Consul cluster back to 64-core servers
   • Disabled the streaming feature for all Consul systems, which brought KV write latency back to 300ms
   • Kept the slow Consul servers from staying elected as leader
   • Redeployed the caching system and fixed its wrong scheduling data and failed deployments
   • Restarted the stopped services at the right capacity levels
   • Let players back in with DNS steering in roughly 10% steps while checking database load, cache performance and stability

   Action items

   • HashiCorp builds laboratory benchmarks and scale tests that reproduce the streaming contention
   • HashiCorp improves the design of streaming so it does not contend under extreme load
   • Compact the BoltDB database with existing BoltDB tooling
   • Remove the circular dependency between the telemetry systems and Consul
   • Extend telemetry with targeted alerts on Consul and BoltDB performance and with visibility into traffic between services and Consul
   • Run the backend in an additional, geographically distinct data center and in multiple availability zones
   • Split critical services into their own dedicated Consul clusters
   • Move data that core services keep directly in the Consul KV store to a more suitable storage system
   • Delete obsolete KV data from Consul
   • Upgrade to a Consul version that replaces BoltDB with bbolt, in Q1
   • Redesign cache deployment so the cache system can be brought up quickly from a standing start
   • Deploy Nomad enhancements that make it easier to turn up large jobs after a long period of unavailability
   • Use mechanisms for faster machine configuration changes
   • Reintroduce streaming only after a new implementation has been tested at Roblox's scale and workload

   Lessons

   Two mostly unrelated problems deep inside Consul, contention in streaming under very high read and write load and BoltDB freelist growth, were hard to diagnose, and that difficulty was the main reason the outage lasted so long. A single Consul cluster serving every workload made the impact larger, and monitoring that depended on Consul hid the cause when it was needed most. Faster hardware did not help and may have hurt, because the contention was worse on NUMA machines. HashiCorp had benchmarked streaming at a similar scale, but the problem needed both a large number of streams and a high churn rate. Restarting a platform from a fully stopped state, including warming caches, took a long time and needed tools built for bootstrapping rather than incremental change.

   View case
5. IL-0271 An empty etcd service entry crashed Discord's Elixir chat nodes, and Discord was down for most users for an hour

   Discord2020sev1

   Symptoms

   On 16 March 2020 at 14:19 PDT most of Discord's Elixir services, which handle real-time connections and chat message processing, crashed at once. Discord lost much of its real-time capacity, the failure cascaded into dependent systems, and most users could not connect for about an hour.

   Service outageHigh error rate

   Trigger

   At 14:18 the Google Cloud managed instance group autoscaler added a node to Discord's API cluster, and the TCP connection of its first announce request to etcd was reset after the request headers were sent but before the body.

   Root cause

   Software defect

   Component

   Coordination and key-value storeService discovery

   Contributing factor

   Automation amplification

   Detection

   At 14:21 several internal alarms fired on a drop in key metrics, along with anomaly alerts for cluster utilization, and a SEV1 incident was declared.

   Remediation

   • Restarted the lost real-time capacity and repaired the failed nodes
   • Throttled connections to Discord to help services recover
   • Stopped anti-entropy repairs on two ScyllaDB clusters so reconnecting users would not starve them of resources
   • Raised the connection throttle step by step to let more users reconnect

   Action items

   • Make the etcd clients send each announce request in a single TCP packet, so headers and body arrive together or not at all
   • Add error handling that ignores services whose key value is corrupt instead of crash-looping
   • File an upstream bug report with the Go project about net/http not checking the body against Content-Length

   Lessons

   Discord's service discovery code handled many etcd failure modes but did not expect a client to write a key without a value because of a transient network error. Go's net/http accepts a request whose body is shorter than its Content-Length header, which breaks the HTTP 1.1 specification and added a failure case nobody had anticipated.

   View case

By root cause4