Cases3 cases

1 to 3 of 3

1. IL-0183 kubeadm's one-year leaf certificates expired, so kube-apiserver could not reach etcd, restarted in a loop and left the cluster unreachable

   ckck8032026

   Symptoms

   The cluster could not be reached from outside. On the control plane node, sudo systemctl status kubelet showed TLS handshake errors such as "http: TLS handshake error from 101.198.0.183:19517: tls: client requested unsupported application protocols" and "http: TLS handshake error from 101.198.0.172:36328: tls: client offered only unsupported versions: [302 301]".

   Control plane unresponsiveContainer crash loop

   Trigger

   The leaf certificates that kubeadm had generated for the cluster, valid for one year by default, expired at 2026-02-20 16:16 UTC.

   Root cause

   Certificate or credential expiry

   Component

   Kubernetes API serveretcdkubeletCalico

   Contributing factor

   Unreviewed defaults

   Detection

   The author could not connect to the cluster from outside and went to the control plane node to read the kubelet logs.

   Remediation

   • renewed all expired certificates with sudo kubeadm certs renew all
   • restarted containerd and the kubelet so the static control plane pods restarted with the new certificates
   • copied the renewed /etc/kubernetes/admin.conf to $HOME/.kube/config

   Lessons

   A kubeadm cluster stops working one year after its leaf certificates were issued unless they are renewed. The first errors seen may be TLS handshake failures or a flag error that does not mention certificates; the decisive message is the x509 expiry in the kube-apiserver log for its connection to etcd. The author recommends checking expiry with kubeadm certs check-expiration and renewing from 30 days before. kubeadm upgrade apply renews the certificates too, so a cluster upgraded at least once a year avoids the problem; a cluster run for a long time without upgrades can renew from cron, for example with 0 3 1 1 * root kubeadm certs renew all && systemctl restart kubelet in /etc/cron.d/kubeadm-cert-renew.

   View case
2. IL-0107 Kyverno stuck in a crash loop behind a webhook with failurePolicy fail blocked every change to Kubernetes Service objects for 88 minutes

   DFDS2022

   Symptoms

   At 11:38 UTC on 3 August 2022 an end-user team told the platform team that applying a Service resource to one of the Kubernetes clusters failed.

   Deploys failingContainer crash loop

   Trigger

   A certificate chain in a third-party application expired, through no action of DFDS, and the outdated Kyverno installation got stuck in a crash loop.

   Root cause

   Certificate or credential expiry

   Component

   Policy engineAdmission webhook

   Contributing factor

   Deferred upgradesUnreviewed defaultsMonitoring blind spotIgnored early signal

   Detection

   An end-user team alerted the platform team at 11:38 UTC that applying a Service resource failed; the platform team started looking into it at 11:50 UTC.

   Remediation

   • reproduced the issue in a sandbox cluster and deployed the fix there first
   • updated Kyverno to a newer Helm chart version in the affected production cluster

   Action items

   • Implement monitoring on Kyverno
   • Create internal documentation on the Kyverno setup
   • Investigate the exact certificate issue in the third-party application
   • Consider changing the failurePolicy on ClusterPolicy resources going forward

   Lessons

   Operational issues should be added to the 'Ops Day to Day' board from the start.

   View case
3. IL-0460 Vault served an expired TLS certificate and almost every GoCardless service pod crash-looped, a 22-minute full outage

   GoCardless2020

   Symptoms

   On Sunday 25 October 2020 the GoCardless API, dashboard, payment pages and partner integrations were all unavailable for 22 minutes, from 07:32 to 07:54 GMT. Automated monitoring paged the on-call engineer at 07:32, a minute after every request to the API had started failing because no healthy backend was left.

   Service outageContainer crash loopTLS certificate errorsCredentials unavailable

   Trigger

   At 07:22 the TLS certificate Vault was still serving expired. cert-manager had already renewed it, but Vault does not load a new certificate without a restart or a SIGHUP.

   Root cause

   Certificate or credential expiry

   Component

   cert-managerSecrets store

   Contributing factor

   Single point of failureMonitoring blind spotUnverified assumptionHarmful manual action

   Detection

   Automated monitoring detected the failing requests and paged the on-call engineer at 07:32. Certificate expiry monitoring existed for the Layer-7 load balancers but did not cover the TCP load balancer that Vault uses.

   Remediation

   • Forced cert-manager to recreate the Vault certificate
   • Restarted the Vault cluster by hand so it loaded the new certificate
   • Deleted some pods of critical workloads to skip their 5-minute exponential backoff

   Action items

   • Check the certificate of the TCP load balancer in front of Vault with the Blackbox exporter, as for the Layer-7 load balancers
   • Deploy the Reloader controller to roll Vault when its certificate data changes
   • Change theatre-envconsul so envconsul no longer manages the child process and only injects secrets at container startup

   Lessons

   A system that almost every service depends on, such as the secrets store, can take everything down at once, and failure scenarios like Vault becoming unreachable had not been tested enough. Assuming the certificate had expired, when it had been renewed but not reloaded, added a slow and unnecessary recreation step to the recovery.

   View case

By root cause1