Cases9 cases

2 to 6 of 9

1. IL-0183 kubeadm's one-year leaf certificates expired, so kube-apiserver could not reach etcd, restarted in a loop and left the cluster unreachable

   ckck8032026

   Symptoms

   The cluster could not be reached from outside. On the control plane node, sudo systemctl status kubelet showed TLS handshake errors such as "http: TLS handshake error from 101.198.0.183:19517: tls: client requested unsupported application protocols" and "http: TLS handshake error from 101.198.0.172:36328: tls: client offered only unsupported versions: [302 301]".

   Control plane unresponsiveContainer crash loop

   Trigger

   The leaf certificates that kubeadm had generated for the cluster, valid for one year by default, expired at 2026-02-20 16:16 UTC.

   Root cause

   Certificate or credential expiry

   Component

   Kubernetes API serveretcdkubeletCalico

   Contributing factor

   Unreviewed defaults

   Detection

   The author could not connect to the cluster from outside and went to the control plane node to read the kubelet logs.

   Remediation

   • renewed all expired certificates with sudo kubeadm certs renew all
   • restarted containerd and the kubelet so the static control plane pods restarted with the new certificates
   • copied the renewed /etc/kubernetes/admin.conf to $HOME/.kube/config

   Lessons

   A kubeadm cluster stops working one year after its leaf certificates were issued unless they are renewed. The first errors seen may be TLS handshake failures or a flag error that does not mention certificates; the decisive message is the x509 expiry in the kube-apiserver log for its connection to etcd. The author recommends checking expiry with kubeadm certs check-expiration and renewing from 30 days before. kubeadm upgrade apply renews the certificates too, so a cluster upgraded at least once a year avoids the problem; a cluster run for a long time without upgrades can renew from cron, for example with 0 3 1 1 * root kubeadm certs renew all && systemctl restart kubelet in /etc/cron.d/kubeadm-cert-renew.

   View case
2. IL-0204 An etcd memory spike overloaded the control plane of a Render Frankfurt cluster, and DNS then failed because CoreDNS ran on the rebooted control plane nodes and could not start without the API server

   Render2024

   Symptoms

   In January 2022 an etcd memory spike overloaded the Kubernetes control plane of one of Render's clusters in Frankfurt. At first only operations that need the control plane failed: new deploys and cron job runs.

   Control plane unresponsiveDeploys failingDNS resolution failuresHigh error rate

   Trigger

   A memory spike in etcd overloaded the cluster's control plane.

   Root cause

   Capacity exhaustion

   Component

   etcdCoreDNSKubernetes API server

   Contributing factor

   Unreviewed defaultsHarmful manual action

   Action items

   • Run CoreDNS on data plane nodes instead of control plane nodes
   • Run etcd on dedicated nodes

   Lessons

   CoreDNS can keep answering from its in-memory cache while the API server is down, but it cannot initialise without it. With CoreDNS on the control plane nodes, a reboot or restart during a control plane outage turned a failure of deploys and cron jobs into a failure of requests to services that were already running. The core problem was that the data plane was not isolated enough from the control plane: even without the reboot, the etcd memory spike alone could have forced CoreDNS to restart.

   View case
3. IL-0077 EKS cluster went read-only for over seven hours after Kyverno ephemeralreports filled the etcd database past its size limit

   Marcin Cuber2024sev1

   Symptoms

   A production EKS cluster on Kubernetes 1.30 went into read-only mode: the API server rejected every request that would change an object, such as creating pods or scaling deployments, and deleting objects to free space did not work either.

   Control plane unresponsiveControl plane lag

   Trigger

   Kyverno 1.12 kept creating ephemeralreports objects that piled up in etcd until the database passed EKS's size limit.

   Root cause

   Software defect

   Component

   etcdKubernetes API serverPolicy engine

   Contributing factor

   No alert on limitLimited debug access

   Detection

   The cluster rejected writes; the audit logs carried the "etcdserver: mvcc: database space exceeded" error.

   Remediation

   • AWS raised the etcd storage limit from 8/10GB to 12GB to unlock writes
   • an AWS engineer deleted the leaked Kyverno ephemeralreports objects directly from etcd

   Action items

   • Monitor the etcd database size with apiserver_storage_size_bytes and never let it reach 8GB
   • Upgrade Kyverno 1.12 to 1.12.4, which fixes the ephemeralreports pile-up, or disable reporting for admission events and tune --aggregationWorkers

   Lessons

   On EKS, once etcd hits its size limit the cluster becomes read-only and only AWS can clean it up, so cleanup scripts must run before the limit is reached. Watch the etcd database size and object counts per resource, especially for controllers that create report objects.

   View case
4. IL-0152 An incorrect configuration applied during maintenance made an internal infrastructure cluster automatically delete the control planes of 200 DOKS clusters in FRA1

   DigitalOcean2024

   Symptoms

   At 13:02 UTC on 2024-01-23 DigitalOcean posted that, as of 12:18 UTC, its engineers were investigating an issue with Kubernetes clusters in the FRA1 region and that users might see errors when communicating with their clusters. The control planes of the affected DOKS (DigitalOcean Kubernetes) clusters had become inaccessible.

   Resources deletedControl plane unresponsive

   Trigger

   An incorrect configuration was applied to an internal infrastructure cluster in FRA1 at 12:21 UTC during a maintenance operation.

   Root cause

   Configuration change

   Component

   etcdKubernetes API server

   Contributing factor

   Automation amplificationNo safeguard on destructive changeExcessive blast radius

   Detection

   Engineers were investigating errors with Kubernetes clusters in FRA1 as of 12:18 UTC and identified the incorrect configuration as the cause at 12:36.

   Remediation

   • restored each affected cluster's etcd from its latest twice-daily backup
   • restored the stateless control plane components after etcd
   • patched the recovery mechanism after unexpected problems during recovery
   • found and fixed an additional small bug during recovery
   • contacted affected customers directly with further instructions, since some might need to reconfigure Kubernetes resources

   Action items

   • Prevent automatic deletion of infrastructure cluster resources under any circumstances in the production environment

   Lessons

   When managed control planes run as resources in another cluster, one configuration mistake in that cluster can delete hundreds of control planes at once through ordinary automation. The etcd backups limited the damage, but their twice-daily schedule set the worst case at 12 hours of lost control plane state, and the recovery tooling itself needed patching during the incident.

   View case
5. IL-0194 After a kube-apiserver restart, pod LIST requests from every pod of a Vector DaemonSet nearly tripled etcd memory and restarted all master nodes

   Palark2023

   Symptoms

   One evening, right after an ordinary restart of kube-apiserver in a customer's production cluster, etcd memory consumption rose almost threefold. The sudden growth led to a cascading restart of all the master nodes.

   Control plane unresponsiveDatabase overload

   Trigger

   An ordinary restart of kube-apiserver, after which every pod of the log collector DaemonSet sent its list requests at the same time.

   Root cause

   Capacity exhaustion

   Component

   etcdKubernetes API serverTelemetry agent

   Contributing factor

   Unknown limitUnreviewed defaults

   Detection

   Memory graphs showed etcd memory on the master nodes almost tripling after a kube-apiserver restart; the audit logs then pointed to one DaemonSet's list requests.

   Remediation

   • created a PriorityLevelConfiguration named limit-list-custom that queues requests (assuredConcurrencyShares 5, 16 queues, handSize 4, queueLengthLimit 50)
   • created a FlowSchema that routes list and get requests on pods from the log collector's service account to that priority level

   Action items

   • Submit a pull request adding resource version support to the Kubernetes Rust library
   • Open a pull request to the Vector log collector to address the startup list requests
   • Apply the same flow-control approach to a similar problem found with the Cilium CNI

   Lessons

   A field selector reduces what kube-apiserver returns, not what it reads from etcd: listing the pods of one node still reads every pod in the cluster. A DaemonSet whose pods all list at startup multiplies that by the node count each time kube-apiserver restarts. Where exact freshness is not needed, resourceVersion=0 lets kube-apiserver answer from its cache; where the client cannot set it, API Priority and Fairness can queue the requests so they reach etcd gradually.

   View case

By root cause6