Incident Library
Applied filters Root causeSoftware defectSymptomDatabase overloadFactorUnpatched known bugClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0587 Elastic Cloud proxies in AWS eu-west-1 flooded an overloaded ZooKeeper and ran out of memory, cutting cluster access
    Elastic2019
    Symptoms

    On 18 January 2019 from 09:09 UTC the leader of the three-node Apache ZooKeeper ensemble behind Elastic Cloud in AWS eu-west-1 (Ireland) ran at heavy CPU load, and its clients lost their connections. From about 10:08 UTC customers in the region had severely degraded access to their Elasticsearch and Kibana deployments for roughly 3 hours, and for about 20 minutes every deployment in the region was unavailable.

    Database overloadService outageOOMKilled
    Trigger

    At 09:09 UTC the leader server of the ZooKeeper ensemble in the eu-west-1 coordination layer hit a high CPU threshold, which caused client disconnections.

    Root cause
    Software defect
    Component
    Coordination and key-value storeService discovery
    Contributing factor
    Retry stormUnpatched known bug
    Detection

    The on-call SREs received alerts about service degradation in the coordination layer at 09:35 UTC, after the high CPU on the ZooKeeper leader had caused client disconnections, and started a production incident.

    Remediation
    • Scaled the proxy layer up and out to take the surging client traffic hitting the load balancers
    • Triggered a leader election in the ZooKeeper ensemble and stabilised it
    Action items
    • Replace Curator's TreeCache with TreeWatcher across all Elasticsearch Service providers and regions
    • Contribute TreeWatcher upstream to the Apache Curator project
    • Roll out a new proxy layer that keeps the routing table in a versioned local file, so it can route traffic while the ZooKeeper connection is lost or flaky
    View case
Check your cluster for this

This case involves Coordination and key-value store and Service discovery. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Software defect1 case

Component
Coordination and key-value storeService discovery
Contributing factor
Unpatched known bugRetry storm
Organisation
Elastic
  1. IL-0587 Elastic Cloud proxies in AWS eu-west-1 flooded an overloaded ZooKeeper and ran out of memory, cutting cluster access
    Elastic2019