Incident Library
Applied filters ComponentCoordination and key-value storeSymptomService outageClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases9 cases

7 to 9 of 9

  1. IL-0587 Elastic Cloud proxies in AWS eu-west-1 flooded an overloaded ZooKeeper and ran out of memory, cutting cluster access
    Elastic2019
    Symptoms

    On 18 January 2019 from 09:09 UTC the leader of the three-node Apache ZooKeeper ensemble behind Elastic Cloud in AWS eu-west-1 (Ireland) ran at heavy CPU load, and its clients lost their connections. From about 10:08 UTC customers in the region had severely degraded access to their Elasticsearch and Kibana deployments for roughly 3 hours, and for about 20 minutes every deployment in the region was unavailable.

    Database overloadService outageOOMKilled
    Trigger

    At 09:09 UTC the leader server of the ZooKeeper ensemble in the eu-west-1 coordination layer hit a high CPU threshold, which caused client disconnections.

    Root cause
    Software defect
    Component
    Coordination and key-value storeService discovery
    Contributing factor
    Retry stormUnpatched known bug
    Detection

    The on-call SREs received alerts about service degradation in the coordination layer at 09:35 UTC, after the high CPU on the ZooKeeper leader had caused client disconnections, and started a production incident.

    Remediation
    • Scaled the proxy layer up and out to take the surging client traffic hitting the load balancers
    • Triggered a leader election in the ZooKeeper ensemble and stabilised it
    Action items
    • Replace Curator's TreeCache with TreeWatcher across all Elasticsearch Service providers and regions
    • Contribute TreeWatcher upstream to the Apache Curator project
    • Roll out a new proxy layer that keeps the routing table in a versioned local file, so it can route traffic while the ZooKeeper connection is lost or flaky
    View case
  2. IL-0276 Chef restarted two of three ZooKeeper nodes at once, and Parse.ly's real-time data stopped for about 3 hours
    Parse.ly2016
    Symptoms

    On Thursday 9 June 2016 at 11:30am ET, real-time data suddenly stopped arriving in Parse.ly's dashboard. Only the data delivery monitors reported a failure; no hardware or machine had failed and no other system reported anything, so the cause was at first unclear.

    Service outage
    Trigger

    A regular Chef run that applied a configuration change restarted the service on two of the three nodes of a ZooKeeper cluster at the same moment.

    Root cause
    Faulty deployment
    Component
    Coordination and key-value store
    Contributing factor
    Unverified assumptionNo staged rollout
    Detection

    Parse.ly's data delivery monitors reported that data had stopped flowing to the dashboard; no other system reported a failure.

    Remediation
    • Cleared a few hours of data from the live systems and cold-started the clusters, taking them offline and bringing them back from scratch
    • Delivered the data from the outage window late through the historical rebuild process
    Action items
    • Separate production configuration changes from production node restarts
    • Always keep a plan B for rapid recovery in systems and processes, instead of treating multi-node failures as impossible
    Lessons

    Automatic routine service restarts are dangerous for services that form a cluster across nodes, and jittering configuration runs only makes simultaneous restarts unlikely, not impossible, once a fleet has hundreds of nodes. A system designed to survive single-node failures still needs a recovery plan for multi-node failures.

    View case
  3. IL-0536 Amazon SimpleDB in US East returned HTTP 500 to nearly all API calls for two hours after its nodes left the cluster
    Amazon Web Services2014
    Symptoms

    From 9:16 PM to 11:16 PM PDT on June 13, Amazon SimpleDB in the US East Region was unavailable to all API calls except a fraction of eventually consistent reads, and SimpleDB API requests returned server-side errors (http response code 500).

    Service outageHigh error rate
    Trigger

    Power was lost to the servers that hosted several SimpleDB storage nodes in a single data center, and those storage nodes became unavailable at the same time.

    Root cause
    Software defect
    Component
    DatabaseCoordination and key-value storeData center power and cooling
    Contributing factor
    Automation amplification
    Remediation
    • Raised the lock service handshake timeout values by hand
    • Restarted a subset of metadata nodes, which could then authorize the storage nodes to rejoin the cluster
    • Throttled CreateDomain and DeleteDomain calls until 1:30 AM while the remaining metadata nodes recovered
    Action items
    • Set a longer lock service handshake timeout in all SimpleDB Regions
    • Make nodes wait and retry handshakes later with a longer timeout instead of removing themselves from the cluster after repeated timeouts
    Lessons

    The short handshake timeout had been chosen to detect replica failures quickly, and AWS concluded it was too low. Nodes removing themselves from the cluster right after several handshake timeouts widened the event and caused the API errors; the nodes should have waited and retried with a longer timeout.

    View case
Check your cluster for this

These 9 cases involve Coordination and key-value store and Service discovery. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause4

Configuration change2 cases

Component
Coordination and key-value store2Database proxy and connection pooler1Database replication1Pod termination1Service discovery1
Contributing factor
Excessive blast radius1Unknown limit1Retry storm1
Organisation
DigitalOcean1Discord1
  1. IL-0499 Discord voice and video calls hung at Awaiting Endpoint for three hours after 17% of sessions stopped at once
    Discord2026
  2. IL-0374 DigitalOcean creates and management actions failed globally when its service discovery datastore lost quorum
    DigitalOcean2024

Cloud provider outage1 case

Component
Coordination and key-value storeSecrets store
Contributing factor
Automation amplification
Organisation
Python Software Foundation
  1. IL-0514 PyPI's backends stopped responding for 7 minutes as rescheduled pods could not start while Consul and Vault were down
    Python Software Foundation2022

Software defect5 cases

Component
Coordination and key-value store5Service discovery3In-memory cache1conntrack table1Container runtime1Database1Data center power and cooling1Elasticsearch or OpenSearch1Linux kernel1
Contributing factor
Automation amplification2Harmful manual action2Monitoring blind spot2Overlapping failures2Retry storm2Change with a delayed effect1Unpatched known bug1Health check tied to a dependency1Single point of failure1
Organisation
Elastic2Amazon Web Services1Discord1Roblox1
  1. IL-0213 Roblox was down for 73 hours because key-value writes on its single Consul cluster slowed from under 300 ms to 2 seconds
    Roblox2021
  2. IL-0271 An empty etcd service entry crashed Discord's Elixir chat nodes, and Discord was down for most users for an hour
    Discord2020
  3. IL-0277 Patching Elastic Cloud's ZooKeeper ensemble in us-east-1 lost quorum, and hosted clusters and Kibana went unreachable
    Elastic2019
  4. IL-0587 Elastic Cloud proxies in AWS eu-west-1 flooded an overloaded ZooKeeper and ran out of memory, cutting cluster access
    Elastic2019
  5. IL-0536 Amazon SimpleDB in US East returned HTTP 500 to nearly all API calls for two hours after its nodes left the cluster
    Amazon Web Services2014

Faulty deployment1 case

Component
Coordination and key-value store
Contributing factor
Unverified assumptionNo staged rollout
Organisation
Parse.ly
  1. IL-0276 Chef restarted two of three ZooKeeper nodes at once, and Parse.ly's real-time data stopped for about 3 hours
    Parse.ly2016