IL-0587 Elastic · 2019-01-18databaseawsnetworking

Elastic Cloud proxies in AWS eu-west-1 flooded an overloaded ZooKeeper and ran out of memory, cutting cluster access

The Elastic Cloud backend in each region has three layers: allocators that host Elasticsearch and Kibana instances, a coordination layer on a 3-node Apache ZooKeeper ensemble, and a proxy and routing layer fronted by load balancers. Each proxy is a direct ZooKeeper client through the Apache Curator framework and uses Curator's TreeCache to mirror the ZooKeeper subtree that holds a znode for every deployment, its nodes running in Docker containers and their properties, with watches keeping the mirror current. Elastic used only the event listener part of TreeCache to maintain its own application-level mirrors.

When a connection is lost and re-established, TreeCache walks the whole mirrored tree and sends one request per znode to reload its data and register a watch, all in one big batch. Some trees have tens of thousands of znodes, and a reconnect during a refresh sends the whole batch again. On 18 January the high CPU on the ZooKeeper leader caused client disconnections, the proxies flooded the servers with refresh batches, filled their incoming request queues and made the GC pauses worse, and the requests waiting in the clients' unbounded outgoing queues led to out of memory errors in the proxies. This chain kept the ensemble from recovering and ended in a complete outage of the region at 12:42 UTC.

During the 20-minute full outage the SREs scaled the proxy layer up and out to take the surging client traffic at the load balancers, triggered a ZooKeeper leader election and stabilised the ensemble; availability returned at 13:03 UTC and the incident was closed after a monitoring period. Elastic had been building TreeWatcher, an API-compatible rewrite of TreeCache, for a few months before the incident. TreeWatcher does not refresh the tree when the ZooKeeper session is resumed, walks the tree only as the server answers so the server sets the pace, and cancels a refresh in progress when the session is lost instead of piling up batches. Its gradual rollout had begun in December 2018, and the cutover of the proxy layer was scheduled for the week of January 21, 2019. By January 24, 2019 TreeWatcher had replaced TreeCache across all Elasticsearch Service providers and regional points of presence.

Severity: –
Scope: Access to Elastic Cloud deployments (Elasticsearch clusters and Kibana instances): severely degraded for about 3 hours, and all deployments in the region unavailable for about 20 minutes
Users affected: Elastic Cloud customers with deployments in AWS eu-west-1
Region: AWS eu-west-1 (Ireland)
Time to detect: 26m
Time to resolve: 3h 54m

Symptoms

On 18 January 2019 from 09:09 UTC the leader of the three-node Apache ZooKeeper ensemble behind Elastic Cloud in AWS eu-west-1 (Ireland) ran at heavy CPU load, and its clients lost their connections. From about 10:08 UTC customers in the region had severely degraded access to their Elasticsearch and Kibana deployments for roughly 3 hours, and for about 20 minutes every deployment in the region was unavailable.

The on-call SREs received alerts about service degradation in the coordination layer at 09:35 UTC. The coordination layer is the ZooKeeper ensemble that holds the system state and the location of every cluster's nodes and Kibana instances. The proxy and routing layer, which sits behind the load balancers and routes HTTP requests and TCP connections to each deployment's nodes, depends on it to keep its routing table consistent, and it went into a degraded state. The proxies saw flaky connections to ZooKeeper: a proxy treats the connection as lost when heartbeat messages do not arrive in time, which also happens when the ZooKeeper server is overloaded and in long GC pauses.

From 09:35 UTC the proxies, which are ZooKeeper clients built on the TreeCache of the Apache Curator framework, ran short of resources as refresh requests piled up, and the proxy layer hit out of memory conditions. The proxy layer deteriorated further after 10:30 UTC, and between 12:42 and 13:02 UTC no cluster in eu-west-1 was available. Cluster availability came back at 13:03 UTC.

Mitigating actions, stopping clients from reconnecting and going through an initialisation phase again, did not let the ZooKeeper quorum stabilise, because reconnecting proxies kept sending large batches of refresh requests to the servers.

Symptom Database overload Service outage OOMKilled

Root cause

Root cause Software defect

Component Coordination and key-value store Service discovery

Contributing factor Retry storm Unpatched known bug

Trigger: At 09:09 UTC the leader server of the ZooKeeper ensemble in the eu-west-1 coordination layer hit a high CPU threshold, which caused client disconnections.

Detection

The on-call SREs received alerts about service degradation in the coordination layer at 09:35 UTC, after the high CPU on the ZooKeeper leader had caused client disconnections, and started a production incident.

Remediation

Scaled the proxy layer up and out to take the surging client traffic hitting the load balancers
Triggered a leader election in the ZooKeeper ensemble and stabilised it

Action items

Replace Curator's TreeCache with TreeWatcher across all Elasticsearch Service providers and regions
Contribute TreeWatcher upstream to the Apache Curator project
Roll out a new proxy layer that keeps the routing table in a versioned local file, so it can route traffic while the ZooKeeper connection is lost or flaky

Lessons

–

Sources

https://www.elastic.co/blog/elastic-cloud-january-18-2019-incident-report · blog · archived copy

Related cases

IL-0213 Roblox was down for 73 hours because key-value writes on its single Consul cluster slowed from under 300 ms to 2 secondsRoblox · 2021
same root cause · shared Coordination and key-value store, Service discovery
IL-0271 An empty etcd service entry crashed Discord's Elixir chat nodes, and Discord was down for most users for an hourDiscord · 2020 · sev1
same root cause · shared Coordination and key-value store, Service discovery
IL-0251 Most of Slack's HAProxy load balancers routed to a stale, shrinking set of webapp backends and served HTTP 503 errorsSlack · 2020
same root cause · shared Service discovery
IL-0277 Patching Elastic Cloud's ZooKeeper ensemble in us-east-1 lost quorum, and hosted clusters and Kibana went unreachableElastic · 2019
same root cause · shared Coordination and key-value store
IL-0536 Amazon SimpleDB in US East returned HTTP 500 to nearly all API calls for two hours after its nodes left the clusterAmazon Web Services · 2014
same root cause · shared Coordination and key-value store

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Coordination and key-value store, Service discovery among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0587
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0587/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.