Incident Library

IL-0277 Elastic · 2019-02-04databaseaws

Patching Elastic Cloud's ZooKeeper ensemble in us-east-1 lost quorum, and hosted clusters and Kibana went unreachable

In each Elastic Cloud region an allocator layer hosts the Elasticsearch and Kibana instances, a coordination layer on a three-node Apache ZooKeeper ensemble holds the system state and the location of every cluster's nodes, and a proxy layer behind load balancers routes traffic by each cluster's virtual host name, reading from a ZooKeeper observer layer that replicates the coordination layer. An ancillary TLS proxying container on every allocator host lets services such as Kibana reach their Elasticsearch cluster securely. During routine maintenance in us-east-1 an engineer replaced ZooKeeper hosts following the documented procedure, met an unfamiliar error, and proceeded because service metrics reported the hosts healthy; those metrics did not show the real state of individual hosts or of the ensemble. Under the load of reconnecting clients the leader shut down at 02:50, and the ensemble could not keep quorum. Because other platform components depend on the coordination layer, losing quorum caused a major disruption, and the proxy layer was impacted as well.

Responders restarted internal services at 03:22 and 06:00 to reduce client connection problems, rolled forward the last ensemble member at 04:07, then cut off client connections at 04:39 and rolled back the third and second members as their hosts became unstable. A runc bug that triggered a kernel OOPS and CPU softlocks when entering the network namespace of the TLS proxying container made two members unresponsive, and their hosts had to be rebooted. At 07:10 a stable quorum formed once client load and operator debugging on the hosts were reduced, and traffic was reintroduced at 07:17. The flood of external reconnections then overloaded the proxy layer and its observer layer; increasing the observers' initLimit setting, which controls how long followers may take to connect and sync to a leader, stabilized them, and at 09:19 the proxy fleet was scaled up and out. Elasticsearch access was restored at 09:28 UTC and the error rate returned to pre-incident levels.

Kibana stayed down longer. The internal TLS proxying service on Kibana hosts could not forward requests, and previously unknown Kibana bugs leaked connections, filling the nf_conn_track table, and amplified HTTP requests back to the load balancing layer. At 12:30 a runbook restarted the internal proxying containers and Kibana instances and applied sysctl limits, and Kibana access was fully restored at 18:44 UTC. Elastic notes that unrelated issues found during the investigation delayed resolution, and that individually successful mitigations were temporarily ineffective at the scale of the failure.

Severity
–
Scope
Elastic Cloud in AWS us-east-1: Elasticsearch deployments partially or completely unavailable from 02:50 to 09:28 UTC, cluster create, modify and delete unavailable while ZooKeeper was down from 02:50 to 07:17 UTC, User Console degraded until 07:17 UTC, Kibana disrupted until 18:44 UTC
Users affected
100% of customers in us-east-1
Region
us-east-1
Time to detect
0m
Time to resolve
15h 54m

Symptoms

On 4 February 2019 at about 02:50 UTC, Elastic Cloud customers with deployments in AWS us-east-1 lost stable access to their Elasticsearch clusters, and an increase in HTTP 5xx errors showed on the load balancer. Deployments were partially or completely unavailable until 09:28 UTC, Kibana access was disrupted for most customers in the region, and the Elastic Cloud User Console timed out when customers tried to view their deployments.

The ZooKeeper ensemble that forms Elastic Cloud's coordination layer was the source. At 02:45, after two of its three members had been replaced with patched hosts, it came under high CPU, IO and memory load with many client connection failures and reconnects, then seemed to settle. At 02:50 it held a leader election; the logs showed the leader overwhelmed by incoming connections and shutting down, and the ensemble could not regain and keep quorum under the request load. Rising latency inside the ensemble caused more client reconnections and made the ZooKeeper observer layer used by the proxies unstable. Customers could not create, modify or delete clusters, and every Elasticsearch cluster in the region saw some failures.

Quorum seemed stable at 03:31 and the replacement of the last member went ahead at 04:07, but at 04:15 that host became unresponsive at the system level. Hosts kept becoming unstable during the rollbacks, and at 05:48 two ensemble members were unresponsive from CPU softlocks caused by runc bugs nobody had seen before. At 06:45 the cause of the host instability was found: when runc entered the network namespace of a TLS proxying service container, it triggered a kernel OOPS and CPU softlocks lasting minutes. Service metrics had reported the hosts as healthy before the maintenance went ahead.

After traffic came back at 07:17, a large wave of external reconnections put high load on the proxy layer and destabilized the proxy ZooKeeper observer layer, so customers still could not connect to their clusters; requests to the proxy layer rose three-fold. Kibana instances then showed much higher CPU than usual from the internal TLS proxying service, which could not forward requests to the proxy layer. Kibana did not close sockets after a failed attempt, so open sockets grew until they filled the nf_conn_track table.

Symptom High error rateService outageIntermittent timeoutsNode kernel panic

Root cause

Root cause Software defect

Component Coordination and key-value storeContainer runtimeLinux kernelElasticsearch or OpenSearchconntrack table
Contributing factor Monitoring blind spotRetry stormHealth check tied to a dependencyOverlapping failuresHarmful manual action
Trigger
A routine patching procedure in AWS us-east-1 replaced the first and second members of the three-node ZooKeeper ensemble with patched hosts between 02:25 and 02:45 UTC, following the same documented process used in the other 12 regions.

Detection

An increase in HTTP 5xx errors on the load balancer was identified at 02:50 UTC, the time the ZooKeeper ensemble lost its leader.

Remediation

Action items

Lessons

Service metrics that reported the ZooKeeper hosts healthy were not enough to show the state of individual hosts or of the ensemble, so they gave a false signal that maintenance could continue. Because the platform depended on the coordination layer, losing its quorum caused a major service disruption in which every Elasticsearch cluster in the region saw some failures, and fixes that worked one by one fell short at the scale of the failure.

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Coordination and key-value store, Container runtime, Linux kernel among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.