Cases23 cases

13 to 17 of 23

1. IL-0172 An AWS Tokyo data centre cooling failure took down 6 of 60 CockroachDB nodes on Kubernetes within 22 minutes, and ranges lost quorum because the Helm-deployed cluster had no availability zone locality configured

   Devsisters2021

   Symptoms

   During a team dinner, a Slack alert reported that one node of the CockroachDB cluster behind Cookie Run: Kingdom had gone down. The DevOps engineers were not worried at first, because CockroachDB is built to survive the loss of some nodes.

   Node NotReadyData lossService outage

   Trigger

   A cooling system failure in an AWS Tokyo region data centre overheated the servers in one availability zone, which held CockroachDB nodes.

   Root cause

   Cloud provider outage

   Component

   Application databaseHelm

   Contributing factor

   Unverified assumptionExcessive blast radius

   Detection

   A Slack alert reported that a database node had gone down.

   Remediation

   • copied the data outside the broken ranges into new tables with INSERT INTO ... SELECT queries that skipped the affected ranges
   • ran debug unsafe-remove-dead-replicas together with Cockroach Labs engineers, recovering one of the two lost data ranges
   • restored the last data range with the storage-layer recovery method from the previous incident

   Action items

   • Configure CockroachDB Locality so data is spread evenly across availability zones
   • Deploy with one chart and a separate values file per availability zone to set Locality
   • Replace nodes by adding Locality-configured nodes, confirming replicas in every zone, then retiring the old nodes

   Lessons

   A replication factor sized for random instance retirements does not protect against a whole availability zone failing; for a game of this scale a multi-AZ placement is required, not optional. Keeping one uniform Helm chart made zone topology hard to set, which is part of why it was left out. A repetitive chart with a values file per zone sets it reliably.

   View case
2. IL-0029 Slow user-facing services from CPU throttling on containers that used far less CPU than their limit

   Buffer2020

   Symptoms

   Services on Buffer's kops-managed cluster on AWS (about 60 nodes and 1500 containers) answered more slowly than they should have. The container throttling metric showed that many containers were being throttled whatever their CPU usage was, including containers that never came close to their CPU limit.

   CPU throttlingHigh latencyNode NotReady

   Trigger

   CPU limits on user-facing services, enforced through CFS quota by a Linux kernel with a bug that throttled containers even when they were well under their limit.

   Root cause

   Software defect

   Component

   Container CPU limitLinux kernelNode operating system image

   Contributing factor

   Unpatched known bugUnverified assumption

   Detection

   While reviewing the container throttling metric, the team noticed many containers being throttled even though their CPU usage stayed far below their limits.

   Remediation

   • removed CPU limits from all services directly or indirectly on the users' critical path
   • moved those services to dedicated tainted nodes and gave them matching tolerations
   • set each service's CPU request to its peak usage over a few months plus a margin of more than 20%, and set memory requests the same way

   Action items

   • Move nodes to a kernel that contains the CFS throttling fix (4.19 or later), which for Buffer means a kops version whose host image is Ubuntu 20.04
   • After the kernel upgrade, check whether throttling is fully gone and update the post

   Lessons

   A CPU limit can slow a service down even when the service uses a small part of it, because of how the kernel enforces CFS quota on older kernels; the throttling metric shows it, the CPU usage graph does not. Upgrading the kernel is the safer fix. Removing limits works for latency, but only together with honest CPU and memory requests, isolation of the unbounded services, and monitoring of node usage, since unbounded containers can starve the kubelet.

   View case
3. IL-0193 A GCP persistent disk degradation took down a Cassandra StatefulSet whose OrderedReady policy blocked recovery, and Metrictank could not reconnect to the recreated pods, causing a 23-hour Graphite query outage

   Grafana Labs2019

   Symptoms

   An alert fired because some Metrictank instances could not drain their write queue, which meant they could not write to Cassandra. Google had not yet published an incident, but a team member's Discord server, also hosted on GCP, was down, which pointed at GCP; Google confirmed an SSD persistent disk incident about 90 minutes after it began.

   High error rateNode NotReadyPods PendingService outage

   Trigger

   A Google Cloud incident in which SSD persistent disks performed so poorly that many I/O operations failed for an extended period.

   Root cause

   Cloud provider outage

   Component

   Persistent volumes and stateful setsApplication databaseNode root disk

   Contributing factor

   Overlapping failuresUnpatched known bugUnverified assumption

   Detection

   An alert that Metrictank write instances could not flush their write queue to Cassandra. Error rates alone rose too slowly to notice quickly, because most queries were served from memory.

   Remediation

   • Added compute nodes hoping new nodes could attach the disks, which did not work
   • Deleted and recreated the Cassandra StatefulSet with an updated pod management policy, so all pods but one started
   • Raised Kafka topic retention to avoid data loss during the long downtime
   • Temporarily added nodes so Metrictank restarts were not slowed by CPU starvation
   • Restarted all Metrictank instances, which took more than 20 hours, so they connected to the new Cassandra IPs

   Action items

   • Re-instantiate the gocql connection in Metrictank when it is detected dead, so it reconnects even when all Cassandra IPs change
   • Fix the Metrictank write-queue deadlock by updating an internal data structure with atomics instead of a lock

   Lessons

   An OrderedReady StatefulSet cannot recover past a single pod whose volume will not attach; when every pod is down, the ordering that protects normal start-up blocks the whole cluster. A client that learns peer addresses only from live peers needs a way to go back to the service name when all peers are replaced at once. A long-tolerated bug with a config workaround can surface at the worst time; this one turned out to have a simple fix. Writing incoming data to a Kafka write-ahead log first meant all data could still be ingested after recovery, and Kafka on non-SSD disks escaped the disk incident. With both Metrictank bugs fixed, the team expects a similar event to take less than 3 hours to recover from instead of 23.

   View case
4. IL-0030 Most nodes of a cluster went NotReady together for 15 minutes while the API server sat behind an AWS ELB

   Zalando2019

   Symptoms

   Fewer than 20% of the nodes in one Zalando cluster were available. Most nodes reported NotReady at the same moment, and only a handful stayed Ready.

   Node NotReady

   Root cause

   Software defect

   Component

   Kubernetes API serverCloud load balancer

   Contributing factor

   Unpatched known bug

   Remediation

   • none; the nodes recovered on their own after about 15 minutes

   View case
5. IL-0035 Worker nodes went NotReady after a new ElastAlert rule without memory limits used up node memory

   NU.nl2019

   Symptoms

   The NU.nl development team got reports that several services on one of its Kubernetes clusters behaved erratically: occasional error pages, slow responses and timeouts. The affected services were mostly internal tools and dashboards.

   Node NotReadyHigh error rateHigh latencyIntermittent timeoutsMonitoring outageOOMKilled

   Trigger

   A new ElastAlert rule, deployed shortly before from the command line, that fired if the Cloudfront-* Elasticsearch indexes stopped receiving documents; those indexes take a few million requests per hour, and the rule made ElastAlert's memory grow without bound.

   Root cause

   Configuration change

   Component

   Container memory limitkubeletPrometheus

   Contributing factor

   No resource requests or limitsUnreviewed defaultsNo alert on limitMonitoring blind spotExcessive blast radius

   Detection

   Users reported error pages, slow responses and timeouts from several services; no alert fired.

   Remediation

   • terminated the NotReady nodes one by one so the auto-scaling group replaced them (the new nodes also went NotReady, but Prometheus and Grafana landed on a stable node)
   • found the ElastAlert pods' memory growth on the Grafana pod memory dashboard
   • fixed the ElastAlert problem that caused the memory growth

   Action items

   • Define alerts on resource usage such as CPU, memory and disk space
   • Keep researching an alerting strategy that combines Elasticsearch, Prometheus Alertmanager and New Relic
   • Consider exporting metrics out of the cluster and running Grafana outside it, on EC2 or a separate Kubernetes cluster
   • Either visualise the important metrics in the ELK stack or make Prometheus and Grafana more available, and improve the metrics strategy
   • Set resource limits for ElastAlert through the Helm values
   • Configure namespaces with default memory requests and limits
   • Isolate customer-facing workloads from build, logging and monitoring workloads (a dedicated node group, or later a separate tools cluster)
   • Deploy everything through automation such as Jenkins pipelines, and consider a GitOps approach
   • Deploy through a pipeline that includes a smoke test step
   • Make Kubernetes and cloud infrastructure work part of the team's sprints, picked up by all team members while pairing with more experienced members, and hold workshops that go deep into specific topics

   Lessons

   Fixing the root cause covers only a small part of what an incident can teach; the chain of smaller failures around it is where most of the learning is. A pod without a memory limit can take whole nodes down, so limits need defaults at the namespace level. Monitoring that runs inside the cluster it watches goes blind with it. Changes installed by hand from the command line leave the rest of the team without a record of what changed.

   View case

By root cause8