Cases1 case

1 to 1 of 1

1. IL-0587 Elastic Cloud proxies in AWS eu-west-1 flooded an overloaded ZooKeeper and ran out of memory, cutting cluster access

   Elastic2019

   Symptoms

   On 18 January 2019 from 09:09 UTC the leader of the three-node Apache ZooKeeper ensemble behind Elastic Cloud in AWS eu-west-1 (Ireland) ran at heavy CPU load, and its clients lost their connections. From about 10:08 UTC customers in the region had severely degraded access to their Elasticsearch and Kibana deployments for roughly 3 hours, and for about 20 minutes every deployment in the region was unavailable.

   Database overloadService outageOOMKilled

   Trigger

   At 09:09 UTC the leader server of the ZooKeeper ensemble in the eu-west-1 coordination layer hit a high CPU threshold, which caused client disconnections.

   Root cause

   Software defect

   Component

   Coordination and key-value storeService discovery

   Contributing factor

   Retry stormUnpatched known bug

   Detection

   The on-call SREs received alerts about service degradation in the coordination layer at 09:35 UTC, after the high CPU on the ZooKeeper leader had caused client disconnections, and started a production incident.

   Remediation

   • Scaled the proxy layer up and out to take the surging client traffic hitting the load balancers
   • Triggered a leader election in the ZooKeeper ensemble and stabilised it

   Action items

   • Replace Curator's TreeCache with TreeWatcher across all Elasticsearch Service providers and regions
   • Contribute TreeWatcher upstream to the Apache Curator project
   • Roll out a new proxy layer that keeps the routing table in a versioned local file, so it can route traffic while the ZooKeeper connection is lost or flaky

   View case

By root cause1