Cases8 cases

5 to 8 of 8

1. IL-0113 The managed GKE control plane of Webhook Relay's main cluster failed because its API server certificates had never been rotated, and a manual credential rotation reported success without taking effect

   Webhook Relay2022

   Symptoms

   At 12:00 EST the health check system reported that the backend APIs, the frontend and the tunnels in the EU region were down. The team first took it for a transient error, such as an interrupted client request or a server evicted for memory or CPU pressure, but the main dashboard was offline as well.

   Service outageControl plane lagControl plane unresponsiveStale or inconsistent API readsCredentials unavailable

   Root cause

   Certificate or credential expiry

   Component

   GKE control planeKubernetes API serverNode poolRBAC and service accounts

   Contributing factor

   Limited debug accessSingle point of failure

   Detection

   The health check system sent a notification that the service was down; the team then found the main dashboard offline.

   Remediation

   • recreated the node pools by hand after the credential rotation failed to recreate them
   • duplicated the main cluster's services into a newly created GKE cluster
   • deleted the Kubernetes Service records in the old cluster so its load balancers could be attached to the new cluster
   • detached the StatefulSet from the old cluster and switched all writes to the main disk
   • exported the data from the temporary ClickHouse instance and inserted it into the main instance

   Action items

   • Change the deployment manifests so services can be switched between clusters quickly without a complicated persistent disk migration

   Lessons

   Recovery of existing infrastructure needs a time limit. Throughout the outage the old cluster looked about to recover, and the reluctance to give up on the salvage effort kept the team from starting over with a new cluster much sooner, which would have been the right call. On older GKE clusters, check the validity of the cluster CA certificate, for example by decoding masterAuth.clusterCaCertificate from gcloud container clusters describe and reading it with openssl x509. GKE's multiple availability zones made no difference in this failure.

   View case
2. IL-0521 AKS pods could not mount volumes and new nodes got no CNI network interface after the cluster service principal expired

   Jenkins project2021

   Symptoms

   From 18:02 UTC on 2 June 2021, the Jenkins infrastructure AKS cluster logged errors about pods that were unable to mount volumes and about scaled-up nodes that were unable to obtain a network virtual interface from the CNI.

   Credentials unavailablePods stuck in ContainerCreating

   Trigger

   The Azure Service Principal credential of the AKS cluster expired at 17:53 UTC on 2 June 2021.

   Root cause

   Certificate or credential expiry

   Component

   Cluster cloud identityPersistent volumes and stateful setsCNI pluginNode pool

   Contributing factor

   Monitoring blind spot

   Detection

   The error behind the failures was found in the AKS UI console while the team investigated the root cause of the LTS 2.289.1 release problem.

   Remediation

   • Renewed the AKS cluster's Azure Service Principal credential, which made AKS restart all nodes automatically

   Action items

   • Add an automatic PagerDuty check that alerts a week before the service principal expires, as is done for certificates
   • Keep a team-level calendar for manual operations such as resetting a credential
   • Consider managed identities, which renew automatically, instead of the service principal

   View case
3. IL-0184 JupyterHub's KubeSpawner could not create containers and kubectl was refused because the API server and other internal certificates of a kubeadm cluster had expired

   dbhang2021

   Symptoms

   KubeSpawner, which starts JupyterHub user containers on Kubernetes, failed to create containers with an error the author had never seen, saying an SSL certificate had expired.

   Control plane unresponsiveDeploys failing

   Root cause

   Certificate or credential expiry

   Component

   Kubernetes API serverkubelet

   Contributing factor

   Unknown limitNo alert on limit

   Detection

   KubeSpawner failed to create a container and logged an SSL certificate verification error.

   Remediation

   • checked certificate expiry with kubeadm alpha certs check-expiration
   • moved the old certificates and /etc/kubernetes/*.conf files to a backup directory
   • regenerated all certificates with kubeadm init phase certs all
   • regenerated the kubeconfig files with kubeadm init phase kubeconfig all
   • copied /etc/kubernetes/admin.conf to ~/.kube/config and restarted kubelet
   • restarted the kube-system containers with docker restart so they picked up the new credentials

   Lessons

   kubeadm clusters run on certificates that expire, and when the API server certificate expires kubectl cannot be used to fix it, so renewal has to happen on the control plane node with kubeadm. After renewal, the kubeconfig files and the running control plane containers must also pick up the new certificates, or kubectl returns Unauthorized.

   View case
4. IL-0460 Vault served an expired TLS certificate and almost every GoCardless service pod crash-looped, a 22-minute full outage

   GoCardless2020

   Symptoms

   On Sunday 25 October 2020 the GoCardless API, dashboard, payment pages and partner integrations were all unavailable for 22 minutes, from 07:32 to 07:54 GMT. Automated monitoring paged the on-call engineer at 07:32, a minute after every request to the API had started failing because no healthy backend was left.

   Service outageContainer crash loopTLS certificate errorsCredentials unavailable

   Trigger

   At 07:22 the TLS certificate Vault was still serving expired. cert-manager had already renewed it, but Vault does not load a new certificate without a restart or a SIGHUP.

   Root cause

   Certificate or credential expiry

   Component

   cert-managerSecrets store

   Contributing factor

   Single point of failureMonitoring blind spotUnverified assumptionHarmful manual action

   Detection

   Automated monitoring detected the failing requests and paged the on-call engineer at 07:32. Certificate expiry monitoring existed for the Layer-7 load balancers but did not cover the TCP load balancer that Vault uses.

   Remediation

   • Forced cert-manager to recreate the Vault certificate
   • Restarted the Vault cluster by hand so it loaded the new certificate
   • Deleted some pods of critical workloads to skip their 5-minute exponential backoff

   Action items

   • Check the certificate of the TCP load balancer in front of Vault with the Blackbox exporter, as for the Layer-7 load balancers
   • Deploy the Reloader controller to roll Vault when its certificate data changes
   • Change theatre-envconsul so envconsul no longer manages the child process and only injects secrets at container startup

   Lessons

   A system that almost every service depends on, such as the secrets store, can take everything down at once, and failure scenarios like Vault becoming unreachable had not been tested enough. Assuming the certificate had expired, when it had been renewed but not reloaded, added a slow and unnecessary recreation step to the recovery.

   View case

By root cause1