Cases3 cases

1 to 3 of 3

1. IL-0285 Clever Cloud's Pulsar brokers stalled in a minor upgrade, then hypervisors overloaded and crashed with kernel panics

   Clever Cloud2024

   Symptoms

   On 2 August 2024, two minutes after Clever Cloud's data team started a minor rolling upgrade of its Apache Pulsar messaging cluster to release 3.3.1, monitoring showed connection problems with the messaging platform. Producing data into the event bus failed, the backlog grew while no new producers or consumers succeeded, and agents on hypervisors, log forwarders on load balancers and deployments were affected. At 10:20 CEST the Pulsar cluster was declared defective.

   Deploys failingSustained high node CPUMemory pressureNode kernel panicOOMKilledAutoscaler misbehaviourMonitoring outage

   Trigger

   At 10:06 CEST the data team began a minor rolling upgrade of Apache Pulsar to release 3.3.1 that also switched on a new balancer for bundle placement: the extensible load manager (loadManagerClassName=org.apache.pulsar.broker.loadbalance.extensions.ExtensibleLoadManagerImpl) with the TransferShedder load shedding strategy instead of the native OverloadShedder.

   Root cause

   Configuration change

   Component

   Message brokerTelemetry agentLinux kernelAutomated capacity manager

   Contributing factor

   No configuration testsMonitoring blind spotIneffective rollbackUnreviewed defaultsAutomation amplificationRetry stormCircular dependency

   Detection

   Monitoring showed connection issues to the messaging platform at 10:08 CEST, two minutes after the upgrade began, and engineers confirmed at 10:09 that producing data into the event bus failed. Monitoring did not catch the first broker's failure to create the system topic, because that broker stayed up.

   Remediation

   • Rolled back the new unload strategy while keeping Pulsar 3.3.1
   • Shut down the client proxies to release the brokers
   • Switched deployment notification events to RabbitMQ as a workaround
   • Stopped all brokers and started only one, which loaded the conflicting system topic so it could be deleted, then started all brokers
   • Restarted the messaging proxies, after which messages flowed again
   • Stopped the watchdog so it would not reload every failed component, and started components one by one
   • Disabled Metrics API ingestion to protect its storage cluster
   • Stopped the long-term storage for observability metrics (the Warp10/FoundationDB cluster) to free resources
   • Put all monitoring instances in all regions into maintenance so they stopped sending events to the orchestrator
   • Killed the overloaded small VMs one by one
   • Flagged 12 hypervisors to drain workload and shut down some load balancers to free CPU and memory
   • Cleaned zombie VMs to speed up recovery
   • Cleared the monitoring alert queue so queued events would not flood the orchestrator
   • Added parked hypervisors to EU-FR-1 to relieve the overloaded ones
   • Removed an overloaded RabbitMQ node from the cluster used by the orchestrator
   • Rolling-restarted every regional L7 load balancer and every L4 load balancer
   • Restarted all HV agents and the add-on provider APIs
   • Destroyed all dangling VMs stuck in the stopping state
   • Re-enabled Metrics API ingestion with a ramp-up QoS policy

   Action items

   • Track KPIs for Pulsar broker and cluster health, hypervisor load, I/O and kernel panic reboots, zombie VMs, RabbitMQ broker and cluster health and deployment queue consumption, each by time of event, detection, decision and resolution
   • Deal with too many monitoring alerts
   • Look at a backup bastion, because a regional bastion is vulnerable
   • Rework the control plane recovery process
   • Rework the correlation between monitoring and orchestration
   • Revisit fault domain separation against multi-tenancy
   • Rethink runtime images and the role of the hypervisor
   • Isolate waved release upgrades of Pulsar
   • Redesign runtime images with better hypervisor integration
   • Regionalise observability for redundancy
   • Set Vector's buffer.max_events and buffer.max_size for small VMs
   • Switch Vector's buffer.type from memory, the default, to disk
   • Change Vector's buffer.when_full from block, the default, to drop_newest
   • Tune journald verbosity and SystemMaxUse on small VMs
   • Add cgroups restrictions for better process control
   • Add I/O restrictions for the VM manager
   • Detect zombie VMs better

   Lessons

   The simulation process that validates non-minor releases was not applied to minor upgrades or to small configuration changes, and this one went straight to production. Running Clever Cloud on Clever Cloud turned into a chicken-and-egg situation: recovery needed the scheduler to carry out large actions while the control plane was itself slowed and the infrastructure had no free resources. Measuring each key indicator by time of event, detection, decision and resolution showed where the team had lacked visibility or reaction and challenged established rules.

   View case
2. IL-0461 Every rollout of Datadog's metrics query service on Kubernetes raised errors that the logs blamed on DNS

   Datadog2022

   Symptoms

   From September 2021, each rollout of an update to Datadog's metrics query service, which serves real-time and historical metric data to the web frontend and to monitor evaluation, came with a rise in errors. Clients retried failed queries, so few errors reached users, but the retries slowed dashboards and monitors. The service's logs showed DNS errors stopping it from connecting to the data stores it depends on.

   High error rateDNS resolution failuresHigh latencyOOMKilledPacket lossStale endpoint routing

   Trigger

   Rollouts of the metrics query service, which delete the old pods. The client behaviour behind the failures came from an August 2021 rollout of the metrics query clients that applied round_robin as the new default gRPC load balancing policy.

   Root cause

   Configuration change

   Component

   conntrack tableCiliumLinux kernelCoreDNSClient-side DNS resolver and cacheexternal-dnsHTTP and RPC client library

   Contributing factor

   Change with a delayed effectRetry stormUnknown limitMonitoring blind spot

   Detection

   Errors rose during rollouts of the metrics query service, and the service's logs showed DNS errors when connecting to its data stores.

   Remediation

   • Raised the memory of the node-local-dns pods from 64 MB to 256 MB, which stopped the OOM kills but not the DNS errors
   • Tested larger instance types and confirmed that scaling up the service's infrastructure resolved the errors
   • Removed the custom gRPC reconnect parameters from the clients to restore the defaults

   Action items

   • Contribute a change to Cilium that adds an unreachable route for a deleted pod's IP so clients get an ICMP error

   Lessons

   The errors looked like DNS but were not. ENA metrics and VPC Flow Logs gave the visibility into VPC connection tracking that host metrics did not, and Cilium, Kubernetes and AWS networking interact in ways that can make reverse path filtering drop packets it should allow. DNS propagation time depends on many settings. The causes were changes Datadog had made itself, to work around a dependency bug, to resolve earlier incidents and to override default reconnect parameters, which looked safe at the time and showed their effects much later, when the link was hard to see.

   View case
3. IL-0277 Patching Elastic Cloud's ZooKeeper ensemble in us-east-1 lost quorum, and hosted clusters and Kibana went unreachable

   Elastic2019

   Symptoms

   On 4 February 2019 at about 02:50 UTC, Elastic Cloud customers with deployments in AWS us-east-1 lost stable access to their Elasticsearch clusters, and an increase in HTTP 5xx errors showed on the load balancer. Deployments were partially or completely unavailable until 09:28 UTC, Kibana access was disrupted for most customers in the region, and the Elastic Cloud User Console timed out when customers tried to view their deployments.

   High error rateService outageIntermittent timeoutsNode kernel panic

   Trigger

   A routine patching procedure in AWS us-east-1 replaced the first and second members of the three-node ZooKeeper ensemble with patched hosts between 02:25 and 02:45 UTC, following the same documented process used in the other 12 regions.

   Root cause

   Software defect

   Component

   Coordination and key-value storeContainer runtimeLinux kernelElasticsearch or OpenSearchconntrack table

   Contributing factor

   Monitoring blind spotRetry stormHealth check tied to a dependencyOverlapping failuresHarmful manual action

   Detection

   An increase in HTTP 5xx errors on the load balancer was identified at 02:50 UTC, the time the ZooKeeper ensemble lost its leader.

   Remediation

   • Restarted internal services in sweeping restarts at 03:22 and 06:00 to reduce client connection problems
   • Isolated the ZooKeeper ensemble by cutting off client connections
   • Rolled back the remaining ensemble member after its host became unstable
   • Rolled back the second ensemble member after its host became unstable
   • Rebooted the ZooKeeper hosts made unresponsive by the runc CPU softlocks
   • Re-established ZooKeeper quorum by removing client load and operator debugging, then reintroduced client load gradually
   • Increased the initLimit setting of the ZooKeeper observer layer used by the proxies
   • Scaled the proxy fleet up and out to handle the increased traffic
   • Applied a runbook that restarted the internal proxying containers and Kibana instances and set sysctl limits

   Action items

   • Reduce the dataset size carried by the ZooKeeper ensembles by over 60%
   • Reduce the number of health checks and intervals needed before a healthy proxy node joins the load balancing layer
   • Stop counting ZooKeeper connectivity in the proxy health checks once a proxy has its configuration and routes
   • Canary the rewritten proxy v2 architecture, which decouples the proxy service from ZooKeeper state, in production regions
   • Improve visibility into ZooKeeper with better logging and metrics for the ZooKeeper processes, the JVM and the container and host systems
   • Make Kibana route discovery more resilient
   • Fix the Kibana issues that amplify internal HTTP requests during availability events
   • Fix the Kibana issues that leak connections
   • Make internal monitoring agents use backoff with jitter
   • Formalize a maintenance matrix with set timeframes per region, so failures of critical services fall in non-critical times
   • Give runbooks for coordination layer maintenance a better decision criteria matrix
   • Set up a feedback loop with support leadership for incident communications through support and the status page

   Lessons

   Service metrics that reported the ZooKeeper hosts healthy were not enough to show the state of individual hosts or of the ensemble, so they gave a false signal that maintenance could continue. Because the platform depended on the coordination layer, losing its quorum caused a major service disruption in which every Elasticsearch cluster in the region saw some failures, and fixes that worked one by one fell short at the scale of the failure.

   View case

By root cause2