Incident Library

Component

Coordination and key-value store

코디네이션 키-값 저장소 · coordination-store

13 cases · 2013 to 2026

Root cause Software defect5Cloud provider outage3Configuration change2Capacity exhaustion1Faulty deployment1DNS or network fault1
Contributing factor Monitoring blind spot5Automation amplification4Harmful manual action3Retry storm3Single point of failure3Unverified assumption2Excessive blast radius2Overlapping failures2Unreviewed defaults1Change with a delayed effect1Ignored early signal1Unpatched known bug1Unknown limit1No headroom1No staged rollout1Health check tied to a dependency1
Symptom Service outage9High error rate6High latency3Database overload2Monitoring outage2Cluster or node provisioning stall1Failover failure1Intermittent connection failures1Intermittent timeouts1Sustained high node CPU1Node kernel panic1OOMKilled1Packet loss1Pods stuck in Terminating1Replication lag1Throughput degradation1Writes rejected1
Organisation Discord2Elastic2Amazon Web Services1AppOptics1CDN771Cloudflare1DigitalOcean1PagerDuty1Parse.ly1Python Software Foundation1Roblox1

Cases, by severity

  1. IL-0271 An empty etcd service entry crashed Discord's Elixir chat nodes, and Discord was down for most users for an hourDiscord · 2020 · sev1

    Root cause Software defect

  2. IL-0362 PagerDuty could not dispatch notifications for 18 minutes when a peering point shared by two AWS regions degradedPagerDuty · 2013 · sev1

    Root cause Cloud provider outage

  3. IL-0499 Discord voice and video calls hung at Awaiting Endpoint for three hours after 17% of sessions stopped at onceDiscord · 2026

    Root cause Configuration change

  4. IL-0288 ZooKeeperWait rose to thousands of seconds on three redundant ClickHouse instances, degrading a CDN77 data pipelineCDN77 · 2026

    Root cause Capacity exhaustion

  5. IL-0374 DigitalOcean creates and management actions failed globally when its service discovery datastore lost quorumDigitalOcean · 2024

    Root cause Configuration change

  6. IL-0514 PyPI's backends stopped responding for 7 minutes as rescheduled pods could not start while Consul and Vault were downPython Software Foundation · 2022

    Root cause Cloud provider outage

  7. IL-0213 Roblox was down for 73 hours because key-value writes on its single Consul cluster slowed from under 300 ms to 2 secondsRoblox · 2021

    Root cause Software defect

  8. IL-0240 Cloudflare API requests failed and the dashboard slowed for hours after the authentication database lost its replicasCloudflare · 2020

    Root cause DNS or network fault

  9. IL-0277 Patching Elastic Cloud's ZooKeeper ensemble in us-east-1 lost quorum, and hosted clusters and Kibana went unreachableElastic · 2019

    Root cause Software defect

  10. IL-0587 Elastic Cloud proxies in AWS eu-west-1 flooded an overloaded ZooKeeper and ran out of memory, cutting cluster accessElastic · 2019

    Root cause Software defect

  11. IL-0588 AppOptics metrics APIs returned 5xx as Kafka partitions stayed leaderless after an AWS power event hit their ZooKeeperAppOptics · 2018

    Root cause Cloud provider outage

  12. IL-0276 Chef restarted two of three ZooKeeper nodes at once, and Parse.ly's real-time data stopped for about 3 hoursParse.ly · 2016

    Root cause Faulty deployment

  13. IL-0536 Amazon SimpleDB in US East returned HTTP 500 to nearly all API calls for two hours after its nodes left the clusterAmazon Web Services · 2014

    Root cause Software defect

View on the graph