Cases3 cases

1 to 3 of 3

1. IL-0204 An etcd memory spike overloaded the control plane of a Render Frankfurt cluster, and DNS then failed because CoreDNS ran on the rebooted control plane nodes and could not start without the API server

   Render2024

   Symptoms

   In January 2022 an etcd memory spike overloaded the Kubernetes control plane of one of Render's clusters in Frankfurt. At first only operations that need the control plane failed: new deploys and cron job runs.

   Control plane unresponsiveDeploys failingDNS resolution failuresHigh error rate

   Trigger

   A memory spike in etcd overloaded the cluster's control plane.

   Root cause

   Capacity exhaustion

   Component

   etcdCoreDNSKubernetes API server

   Contributing factor

   Unreviewed defaultsHarmful manual action

   Action items

   • Run CoreDNS on data plane nodes instead of control plane nodes
   • Run etcd on dedicated nodes

   Lessons

   CoreDNS can keep answering from its in-memory cache while the API server is down, but it cannot initialise without it. With CoreDNS on the control plane nodes, a reboot or restart during a control plane outage turned a failure of deploys and cron jobs into a failure of requests to services that were already running. The core problem was that the data plane was not isolated enough from the control plane: even without the reboot, the etcd memory spike alone could have forced CoreDNS to restart.

   View case
2. IL-0060 Kubernetes 1.24 upgrade removed the master node label that Calico route reflectors selected on, dropping all pod routes and taking Reddit down for 314 minutes

   Reddit2023

   Symptoms

   About two minutes after an engineer started the Kubernetes 1.23 to 1.24 upgrade of the cluster running Reddit's legacy stack, just after 19:00 UTC, the site stopped. The CDN edge request graph, which is kept separate from the cluster, showed traffic by status collapsing. Users saw the error page, a homepage with no posts, or plain errors.

   Service outageHigh error rateDNS resolution failuresMonitoring outagePods stuck in ContainerCreatingControl plane lag

   Trigger

   An in-place upgrade of the legacy cluster from Kubernetes 1.23 to 1.24; in 1.24 the node-role.kubernetes.io/master label is removed even from running clusters.

   Root cause

   Breaking change in an upgrade

   Component

   CalicoKubernetes API serverAdmission webhooketcd

   Contributing factor

   Configuration not in codeUnrepresentative stagingUntested backupMonitoring blind spotSingle point of failure

   Detection

   The site broke about two minutes into the upgrade; an incident was opened immediately and responders were on the call within three minutes, working from CDN edge graphs because the cluster's own metrics were gone.

   Remediation

   • deleted the Open Policy Agent webhook configurations to stop API server timeouts
   • terminated all worker nodes of the cluster
   • restored the control plane from the pre-upgrade backup onto the first control plane node
   • switched off the cluster autoscaler until the control plane was stable
   • generated new TLS certificates so replacement control plane nodes could join etcd
   • brought traffic back in steps from 1% to 100%
   • restarted legacy services that had been stopped by hand

   Action items

   • Codify cluster configuration so every choice has a record, and document in detail what cannot be codified
   • Standardise components across clusters and replace bespoke configurations
   • Build tooling to manage the lifecycle of whole clusters so they can be recreated or replicated
   • Add pre-production clusters that can be tested with confidence
   • Improve the backup restore process and make cluster upgrades safe

   Lessons

   Configuration that exists only in a live cluster, edited by hand through a special tool, can be forgotten, as the route reflector setup was, and then break on an upgrade nobody connects it to. A Kubernetes upgrade cannot be reverted without a restore, so the restore procedure has to be kept current and practised. Consistent, codified clusters are what make upgrades testable; bespoke clusters keep producing outages.

   View case
3. IL-0075 Kubernetes API servers pegged on CPU when the cluster grew to about 290 nodes, traced to Datadog agents polling the API

   Civis Analytics2019

   Symptoms

   After Civis released its new script infrastructure on Kubernetes internally in September 2018, developers reported that kubectl commands were slow, and users intermittently could not start notebooks and services.

   Control plane lagControl plane unresponsiveHigh latencyHigh error rateDNS resolution failures

   Trigger

   Migrating clients' Python, R and container scripts from the in-house Bocce system to Kubernetes (the project called Croquet) pushed the cluster to about 290 EC2 instances under a large script load.

   Root cause

   Capacity exhaustion

   Component

   Kubernetes API serverTelemetry agentetcd

   Contributing factor

   Unknown limitUnverified assumptionMonitoring blind spotNo capacity planning

   Detection

   Developer reports of slow kubectl commands and user reports of notebooks and services failing to start, then high CPU on the master nodes.

   Remediation

   • moved internal users back to the old Bocce infrastructure
   • resized the three master nodes from m4.large to m4.4xlarge
   • changed the Datadog setup so per-node agents no longer polled the API server for service metadata
   • raised max-requests-inflight for non-mutating requests from 400 to 1200
   • grew the etcd EBS volumes from 20 to 100 GB (100 to 300 IOPS)
   • switched client scripts from the ClusterFirst DNS policy to Default

   Action items

   • Upgrade etcd from 2.2.1 to 3.x once Kops supports a safe migration
   • Solve the in-cluster DNS timeouts so client scripts can talk to other workloads in the cluster

   Lessons

   Kubernetes can reach thousands of nodes only after tuning, and third-party DaemonSets are a common cause of control plane load: an agent on every node that polls or watches the API server multiplies with the cluster size. Datadog, kube2iam and the fluentd metadata filter all showed this. A small configuration change often fixes it, but finding it took load testing and the API server access logs.

   View case

By root cause2