Cases2 cases

1 to 2 of 2

1. IL-0277 Patching Elastic Cloud's ZooKeeper ensemble in us-east-1 lost quorum, and hosted clusters and Kibana went unreachable

   Elastic2019

   Symptoms

   On 4 February 2019 at about 02:50 UTC, Elastic Cloud customers with deployments in AWS us-east-1 lost stable access to their Elasticsearch clusters, and an increase in HTTP 5xx errors showed on the load balancer. Deployments were partially or completely unavailable until 09:28 UTC, Kibana access was disrupted for most customers in the region, and the Elastic Cloud User Console timed out when customers tried to view their deployments.

   High error rateService outageIntermittent timeoutsNode kernel panic

   Trigger

   A routine patching procedure in AWS us-east-1 replaced the first and second members of the three-node ZooKeeper ensemble with patched hosts between 02:25 and 02:45 UTC, following the same documented process used in the other 12 regions.

   Root cause

   Software defect

   Component

   Coordination and key-value storeContainer runtimeLinux kernelElasticsearch or OpenSearchconntrack table

   Contributing factor

   Monitoring blind spotRetry stormHealth check tied to a dependencyOverlapping failuresHarmful manual action

   Detection

   An increase in HTTP 5xx errors on the load balancer was identified at 02:50 UTC, the time the ZooKeeper ensemble lost its leader.

   Remediation

   • Restarted internal services in sweeping restarts at 03:22 and 06:00 to reduce client connection problems
   • Isolated the ZooKeeper ensemble by cutting off client connections
   • Rolled back the remaining ensemble member after its host became unstable
   • Rolled back the second ensemble member after its host became unstable
   • Rebooted the ZooKeeper hosts made unresponsive by the runc CPU softlocks
   • Re-established ZooKeeper quorum by removing client load and operator debugging, then reintroduced client load gradually
   • Increased the initLimit setting of the ZooKeeper observer layer used by the proxies
   • Scaled the proxy fleet up and out to handle the increased traffic
   • Applied a runbook that restarted the internal proxying containers and Kibana instances and set sysctl limits

   Action items

   • Reduce the dataset size carried by the ZooKeeper ensembles by over 60%
   • Reduce the number of health checks and intervals needed before a healthy proxy node joins the load balancing layer
   • Stop counting ZooKeeper connectivity in the proxy health checks once a proxy has its configuration and routes
   • Canary the rewritten proxy v2 architecture, which decouples the proxy service from ZooKeeper state, in production regions
   • Improve visibility into ZooKeeper with better logging and metrics for the ZooKeeper processes, the JVM and the container and host systems
   • Make Kibana route discovery more resilient
   • Fix the Kibana issues that amplify internal HTTP requests during availability events
   • Fix the Kibana issues that leak connections
   • Make internal monitoring agents use backoff with jitter
   • Formalize a maintenance matrix with set timeframes per region, so failures of critical services fall in non-critical times
   • Give runbooks for coordination layer maintenance a better decision criteria matrix
   • Set up a feedback loop with support leadership for incident communications through support and the status page

   Lessons

   Service metrics that reported the ZooKeeper hosts healthy were not enough to show the state of individual hosts or of the ensemble, so they gave a false signal that maintenance could continue. Because the platform depended on the coordination layer, losing its quorum caused a major service disruption in which every Elasticsearch cluster in the region saw some failures, and fixes that worked one by one fell short at the scale of the failure.

   View case
2. IL-0520 logs.cloud.gov trapped users in a login redirect loop because its full Redis session store stopped saving sessions

   cloud.gov2016

   Symptoms

   At 7:12pm Eastern Time on 5 July 2016, logs.cloud.gov, the cloud.gov log search service, sent users into a login redirect loop, so they could not log in to the service.

   Service outageWrites rejected

   Root cause

   Capacity exhaustion

   Component

   In-memory cacheElasticsearch or OpenSearch

   Contributing factor

   Overlapping failuresMonitoring blind spot

   Remediation

   • Moved Redis on the logsearch service to a much larger memory instance so it could keep logs while Elasticsearch was down
   • Corrected the Elasticsearch memory limits and the /tmp noexec hardening setting and redeployed the cluster
   • Redeployed the Kibana instance and verified the logs once the cluster deployment was complete

   Action items

   • Add a failure alert that checks whether logs are coming through
   • Make Redis auto-scaling or highly available so that failures in more fragile downstream components do not cause data loss
   • Extend the incident response plan to start with events that are not security-specific
   • Create a chaos bot for the logs.cloud.gov service

   View case

By root cause2