Cases8 cases

5 to 8 of 8

1. IL-0193 A GCP persistent disk degradation took down a Cassandra StatefulSet whose OrderedReady policy blocked recovery, and Metrictank could not reconnect to the recreated pods, causing a 23-hour Graphite query outage

   Grafana Labs2019

   Symptoms

   An alert fired because some Metrictank instances could not drain their write queue, which meant they could not write to Cassandra. Google had not yet published an incident, but a team member's Discord server, also hosted on GCP, was down, which pointed at GCP; Google confirmed an SSD persistent disk incident about 90 minutes after it began.

   High error rateNode NotReadyPods PendingService outage

   Trigger

   A Google Cloud incident in which SSD persistent disks performed so poorly that many I/O operations failed for an extended period.

   Root cause

   Cloud provider outage

   Component

   Persistent volumes and stateful setsApplication databaseNode root disk

   Contributing factor

   Overlapping failuresUnpatched known bugUnverified assumption

   Detection

   An alert that Metrictank write instances could not flush their write queue to Cassandra. Error rates alone rose too slowly to notice quickly, because most queries were served from memory.

   Remediation

   • Added compute nodes hoping new nodes could attach the disks, which did not work
   • Deleted and recreated the Cassandra StatefulSet with an updated pod management policy, so all pods but one started
   • Raised Kafka topic retention to avoid data loss during the long downtime
   • Temporarily added nodes so Metrictank restarts were not slowed by CPU starvation
   • Restarted all Metrictank instances, which took more than 20 hours, so they connected to the new Cassandra IPs

   Action items

   • Re-instantiate the gocql connection in Metrictank when it is detected dead, so it reconnects even when all Cassandra IPs change
   • Fix the Metrictank write-queue deadlock by updating an internal data structure with atomics instead of a lock

   Lessons

   An OrderedReady StatefulSet cannot recover past a single pod whose volume will not attach; when every pod is down, the ordering that protects normal start-up blocks the whole cluster. A client that learns peer addresses only from live peers needs a way to go back to the service name when all peers are replaced at once. A long-tolerated bug with a config workaround can surface at the worst time; this one turned out to have a simple fix. Writing incoming data to a Kafka write-ahead log first meant all data could still be ingested after recovery, and Kafka on non-SSD disks escaped the disk incident. With both Metrictank bugs fixed, the team expects a similar event to take less than 3 hours to recover from instead of 23.

   View case
2. IL-0066 Broken ConfigMap change lay dormant for weeks, then took down Pusher's edge proxy when a cluster upgrade replaced its pods

   Pusher2019

   Symptoms

   During a cluster upgrade, all traffic to Pusher's products stopped. The speaker recalls that requests were failing with 500 errors.

   Service outageHigh error rate

   Trigger

   A cluster upgrade replaced nodes, so the bridge pods restarted and loaded a ConfigMap that had been changed three or four weeks earlier.

   Root cause

   Configuration change

   Component

   ConfigMaps and Secrets

   Contributing factor

   Change with a delayed effectUnverified assumption

   Action items

   • Build and run Wave so that changes to ConfigMaps and Secrets roll out new pods immediately

   Lessons

   A ConfigMap change does not restart the pods that use it, so for an application that cannot reload its configuration the change takes effect only when pods are next replaced, possibly weeks later during unrelated work. Tie configuration to the pod template so a bad change fails at the moment it is made.

   View case
3. IL-0064 CPU-heavy pods packed onto one Google Cloud node drove it to kernel panics, and the crash moved from node to node until the site was down

   Moonlight2019

   Symptoms

   On Friday, January 18, Moonlight's API began returning occasional errors that looked like connection problems with its Redis database. The API checks Redis on every authenticated request to validate the session. At the same time the Kubernetes monitoring reported some nodes and pods as unresponsive.

   High error rateIntermittent connection failuresNode NotReadyService outageSustained high node CPU

   Trigger

   The Kubernetes scheduler put several CPU-heavy pods on the same node, including all three replicas of the website and the server-side rendering pods.

   Root cause

   Capacity exhaustion

   Component

   Pod scheduling and placementLinux kernelApplication database

   Contributing factor

   Single point of failureUnverified assumptionAutomation amplification

   Detection

   API errors that looked like Redis connectivity problems, and Kubernetes monitoring that showed nodes and pods as unresponsive. Google Cloud Support later found the link between sustained 100% CPU and kernel panics.

   Remediation

   • added anti-affinity rules to all major deployments so their pods spread across nodes

   Action items

   • Keep working with Google Cloud support to find and fix the cause of the node kernel panics

   Lessons

   Three replicas give no fault tolerance if the scheduler places them all on one node; anti-affinity rules make the spread explicit. A node failure that comes from the workload itself moves with the workload when the scheduler reschedules it.

   View case
4. IL-0075 Kubernetes API servers pegged on CPU when the cluster grew to about 290 nodes, traced to Datadog agents polling the API

   Civis Analytics2019

   Symptoms

   After Civis released its new script infrastructure on Kubernetes internally in September 2018, developers reported that kubectl commands were slow, and users intermittently could not start notebooks and services.

   Control plane lagControl plane unresponsiveHigh latencyHigh error rateDNS resolution failures

   Trigger

   Migrating clients' Python, R and container scripts from the in-house Bocce system to Kubernetes (the project called Croquet) pushed the cluster to about 290 EC2 instances under a large script load.

   Root cause

   Capacity exhaustion

   Component

   Kubernetes API serverTelemetry agentetcd

   Contributing factor

   Unknown limitUnverified assumptionMonitoring blind spotNo capacity planning

   Detection

   Developer reports of slow kubectl commands and user reports of notebooks and services failing to start, then high CPU on the master nodes.

   Remediation

   • moved internal users back to the old Bocce infrastructure
   • resized the three master nodes from m4.large to m4.4xlarge
   • changed the Datadog setup so per-node agents no longer polled the API server for service metadata
   • raised max-requests-inflight for non-mutating requests from 400 to 1200
   • grew the etcd EBS volumes from 20 to 100 GB (100 to 300 IOPS)
   • switched client scripts from the ClusterFirst DNS policy to Default

   Action items

   • Upgrade etcd from 2.2.1 to 3.x once Kops supports a safe migration
   • Solve the in-cluster DNS timeouts so client scripts can talk to other workloads in the cluster

   Lessons

   Kubernetes can reach thousands of nodes only after tuning, and third-party DaemonSets are a common cause of control plane load: an agent on every node that polls or watches the API server multiplies with the cluster size. Datadog, kube2iam and the fluentd metadata filter all showed this. A small configuration change often fixes it, but finding it took load testing and the API server access logs.

   View case

By root cause5