Cases6 cases

1 to 5 of 6

1. IL-0183 kubeadm's one-year leaf certificates expired, so kube-apiserver could not reach etcd, restarted in a loop and left the cluster unreachable

   ckck8032026

   Symptoms

   The cluster could not be reached from outside. On the control plane node, sudo systemctl status kubelet showed TLS handshake errors such as "http: TLS handshake error from 101.198.0.183:19517: tls: client requested unsupported application protocols" and "http: TLS handshake error from 101.198.0.172:36328: tls: client offered only unsupported versions: [302 301]".

   Control plane unresponsiveContainer crash loop

   Trigger

   The leaf certificates that kubeadm had generated for the cluster, valid for one year by default, expired at 2026-02-20 16:16 UTC.

   Root cause

   Certificate or credential expiry

   Component

   Kubernetes API serveretcdkubeletCalico

   Contributing factor

   Unreviewed defaults

   Detection

   The author could not connect to the cluster from outside and went to the control plane node to read the kubelet logs.

   Remediation

   • renewed all expired certificates with sudo kubeadm certs renew all
   • restarted containerd and the kubelet so the static control plane pods restarted with the new certificates
   • copied the renewed /etc/kubernetes/admin.conf to $HOME/.kube/config

   Lessons

   A kubeadm cluster stops working one year after its leaf certificates were issued unless they are renewed. The first errors seen may be TLS handshake failures or a flag error that does not mention certificates; the decisive message is the x509 expiry in the kube-apiserver log for its connection to etcd. The author recommends checking expiry with kubeadm certs check-expiration and renewing from 30 days before. kubeadm upgrade apply renews the certificates too, so a cluster upgraded at least once a year avoids the problem; a cluster run for a long time without upgrades can renew from cron, for example with 0 3 1 1 * root kubeadm certs renew all && systemctl restart kubelet in /etc/cron.d/kubeadm-cert-renew.

   View case
2. IL-0232 Calico pods in the codfw WikiKube cluster hit their resource limits and crash-looped, taking nodes out one by one

   Wikimedia Foundation2024

   Symptoms

   At 05:01 UTC on 9 October 2024 a page fired for GatewayBackendErrorsHigh, elevated 5xx errors from page-analytics_cluster in codfw, followed by a growing number of KubernetesCalicoDown alerts for WikiKube nodes not running the calico-node pod and by MediaWikiMemcachedHighErrorRate. For about an hour uncached MediaWiki requests, such as those of logged-in users, edits and API calls, saw high latency or errors.

   High error rateHigh latencyContainer crash loopProbe failuresControl plane lag

   Trigger

   An operator used the new mwscript-k8s utility to run a batch of video transcode maintenance, the TimedMediaHandler requeueTranscodes.php script. mwscript-k8s deploys a whole helm release for every script invocation, so the batch created close to 2k Calico NetworkPolicies along with many other resources over several hours.

   Root cause

   Capacity exhaustion

   Component

   CalicoHelmJobs and CronJobs

   Contributing factor

   Change with a delayed effectNo runbook

   Detection

   Automated monitoring detected the issue and paged the on-call SRE: GatewayBackendErrorsHigh at 05:01:36 UTC, 16 KubernetesCalicoDown alerts at 05:03:32 UTC and MediaWikiMemcachedHighErrorRate at 05:04:15 UTC.

   Remediation

   • Raised the resource limits of the Calico and Typha pods and applied the change with helmfile, which redeployed all Calico pods

   Action items

   • Stop mwscript-k8s from creating so many resources

   Lessons

   The incident came close to a full outage. Automated monitoring caught it and many responders found the cause, but end users were affected and responders were unsure whether they could switch over to eqiad. The problem was noticed just before the hand-off from Americas to EMEA SRE shifts, a period with a long-known lack of coverage, and it helped that the transcodes were not queued on a Friday evening in the US.

   View case
3. IL-0087 After a cold reboot of RKE2 nodes, pods could not get IP addresses because Multus could not reach the Kubernetes API, and further reboots spread the failure to every node

   Fyra Labs2023

   Symptoms

   At around 2 AM ICT on 2023-04-09 the team saw unusual CPU use on one cluster node, which throttled every other service on it and made them fail. They presumed an upstream container runtime bug in containerd and runc and decided to fully reboot the affected nodes, taking the cluster down for maintenance on the assumption that it was not in active use.

   Sustained high node CPUService outage

   Trigger

   A full cold reboot of the nodes that showed unusual CPU use, followed by upgrades and cold reboots of the other nodes.

   Root cause

   unknown

   Component

   Multus meta-CNICalicoContainer runtime

   Contributing factor

   Harmful manual actionUnverified assumption

   Detection

   The team saw unusual CPU use on a node around 2 AM ICT and failing services on the rebooted nodes around 2:30 ICT; developers reported the Terra repositories down at 6:31 ICT.

   Remediation

   • uninstalled Multus and Calico and replaced them with the Cilium CNI
   • rolled back most cluster settings except the CNI

   Lessons

   Rebooting nodes to clear one problem can expose another; here a CNI chain could not reach the API server after a cold start. Repeating the same reboot on more nodes while the first failure was not understood spread the outage from a few nodes to the whole cluster. The team also found that Cilium's real resource use in their cluster was acceptable, contrary to the benchmark they had relied on.

   View case
4. IL-0060 Kubernetes 1.24 upgrade removed the master node label that Calico route reflectors selected on, dropping all pod routes and taking Reddit down for 314 minutes

   Reddit2023

   Symptoms

   About two minutes after an engineer started the Kubernetes 1.23 to 1.24 upgrade of the cluster running Reddit's legacy stack, just after 19:00 UTC, the site stopped. The CDN edge request graph, which is kept separate from the cluster, showed traffic by status collapsing. Users saw the error page, a homepage with no posts, or plain errors.

   Service outageHigh error rateDNS resolution failuresMonitoring outagePods stuck in ContainerCreatingControl plane lag

   Trigger

   An in-place upgrade of the legacy cluster from Kubernetes 1.23 to 1.24; in 1.24 the node-role.kubernetes.io/master label is removed even from running clusters.

   Root cause

   Breaking change in an upgrade

   Component

   CalicoKubernetes API serverAdmission webhooketcd

   Contributing factor

   Configuration not in codeUnrepresentative stagingUntested backupMonitoring blind spotSingle point of failure

   Detection

   The site broke about two minutes into the upgrade; an incident was opened immediately and responders were on the call within three minutes, working from CDN edge graphs because the cluster's own metrics were gone.

   Remediation

   • deleted the Open Policy Agent webhook configurations to stop API server timeouts
   • terminated all worker nodes of the cluster
   • restored the control plane from the pre-upgrade backup onto the first control plane node
   • switched off the cluster autoscaler until the control plane was stable
   • generated new TLS certificates so replacement control plane nodes could join etcd
   • brought traffic back in steps from 1% to 100%
   • restarted legacy services that had been stopped by hand

   Action items

   • Codify cluster configuration so every choice has a record, and document in detail what cannot be codified
   • Standardise components across clusters and replace bespoke configurations
   • Build tooling to manage the lifecycle of whole clusters so they can be recreated or replicated
   • Add pre-production clusters that can be tested with confidence
   • Improve the backup restore process and make cluster upgrades safe

   Lessons

   Configuration that exists only in a live cluster, edited by hand through a special tool, can be forgotten, as the route reflector setup was, and then break on an upgrade nobody connects it to. A Kubernetes upgrade cannot be reverted without a restore, so the restore procedure has to be kept current and practised. Consistent, codified clusters are what make upgrades testable; bespoke clusters keep producing outages.

   View case
5. IL-0139 Pods on GKE clusters with Calico network policy got stuck Terminating or Pending after upgrading to 1.22 or later because of a race condition in the Calico component

   Google Cloud2022

   Symptoms

   After upgrading to GKE 1.22 or later, clusters that use Calico network policy could have pods stuck in Terminating or Pending. Google Cloud posted the incident at 16:01 US/Pacific on 2022-09-15 as pods stuck terminating; from the 2022-09-21 update the description also named pods stuck Pending.

   Pods stuck in TerminatingPods Pending

   Trigger

   Upgrading a GKE cluster that uses Calico network policy to version 1.22 or later.

   Root cause

   Software defect

   Component

   CalicokubeletPod termination

   Contributing factor

   –

   Remediation

   • released a fix in GKE 1.24.4-gke.800 and made it available in 1.23 and 1.22
   • offered an internal patch through Google Cloud Support for affected customers
   • advised customers to restart the kubelet and calico-node to get stuck pods unstuck
   • advised customers without a strong need for Calico to disable Calico network policy

   Action items

   • Upgrade clusters on the Rapid, Regular and Stable release channels running 1.22 or 1.23 to the fixed version automatically

   Lessons

   A race in the Calico component can leave pods stuck Terminating, with FailedKillPod and KillPodSandbox errors that point at the network plugin, not at the workload. Restarting the kubelet and calico-node frees the stuck pods but does not remove the race; the fixed GKE version or Google's internal patch does. Clusters that use the cluster autoscaler are more likely to hit it.

   View case

By root cause6