Cases26 cases

19 to 23 of 26

1. IL-0184 JupyterHub's KubeSpawner could not create containers and kubectl was refused because the API server and other internal certificates of a kubeadm cluster had expired

   dbhang2021

   Symptoms

   KubeSpawner, which starts JupyterHub user containers on Kubernetes, failed to create containers with an error the author had never seen, saying an SSL certificate had expired.

   Control plane unresponsiveDeploys failing

   Root cause

   Certificate or credential expiry

   Component

   Kubernetes API serverkubelet

   Contributing factor

   Unknown limitNo alert on limit

   Detection

   KubeSpawner failed to create a container and logged an SSL certificate verification error.

   Remediation

   • checked certificate expiry with kubeadm alpha certs check-expiration
   • moved the old certificates and /etc/kubernetes/*.conf files to a backup directory
   • regenerated all certificates with kubeadm init phase certs all
   • regenerated the kubeconfig files with kubeadm init phase kubeconfig all
   • copied /etc/kubernetes/admin.conf to ~/.kube/config and restarted kubelet
   • restarted the kube-system containers with docker restart so they picked up the new credentials

   Lessons

   kubeadm clusters run on certificates that expire, and when the API server certificate expires kubectl cannot be used to fix it, so renewal has to happen on the control plane node with kubeadm. After renewal, the kubeconfig files and the running control plane containers must also pick up the new certificates, or kubectl returns Unauthorized.

   View case
2. IL-0458 Toolforge Kubernetes API server failed to start after a restart, logging "etcd cluster is unavailable or misconfigured"

   Wikimedia Foundation2019

   Symptoms

   On 10 September 2019 the Kubernetes API server of Wikimedia's Toolforge cluster did not come back after the maintain-kubeusers process restarted it at 18:30. To monitoring and to users the whole cluster looked down: Toolforge users could not launch tools with the webservice command on the Kubernetes backend or use kubectl, and an Icinga check against the "All k8s nodes healthy" endpoint started alerting at 18:45.

   Control plane unresponsiveService outageTLS certificate errors

   Trigger

   At 18:30 the maintain-kubeusers process restarted the Kubernetes API server, which forced it to reconnect to etcd with a CA certificate that Puppet changes the night before had left mismatched with the etcd servers.

   Root cause

   Configuration change

   Component

   Kubernetes API serveretcdConfiguration management tool

   Contributing factor

   Change with a delayed effectHarmful manual actionNo runbook

   Detection

   Automated monitoring: an Icinga alert on the toolschecker check http://checker.tools.wmflabs.org/k8s/nodes/ready, about "All k8s nodes healthy", fired at 18:45.

   Remediation

   • Rebooted the Kubernetes master and the etcd cluster, which did not help
   • Took a backup of etcd and reduced the etcd cluster to one node to rule out raft election flaps
   • Brought etcd back to full capacity through a full disaster recovery from the backup
   • Swapped the API server's CA certificate for the one from the etcd servers, which restored services until Puppet reverted it
   • Placed the etcd Puppet CA certificate in /var/lib/puppet/client/ssl/certs/ca.pem so Puppet kept the right CA

   Action items

   • Document the etcd cluster processes
   • Upgrade Toolforge Kubernetes to improve etcd and Kubernetes error reporting and stop needing an API server restart to add a tool
   • Make kube2proxy more resilient to API server failures
   • Audit the Puppet CA certificates so the tools project has no more than one
   • Make the Kubernetes control plane highly available

   Lessons

   Components that often misbehave, such as etcd, drew the responders to the wrong fix, and working on etcd instead of the CA certificates made the outage worse. Puppet-managed certificates are risky, and very old Kubernetes versions log errors that are hard to read. The failure was hard to find because it surfaced at an API server restart rather than at the Puppet change, because the etcd cluster had long-standing undocumented quirks, and because no disaster recovery or node replacement procedure was written down for etcd.

   View case
3. IL-0008 An unresponsive OPA validating webhook crash-looped the API server during a GKE master upgrade, and node auto-repair then recreated every node

   Jetstack2019

   Symptoms

   Jetstack was upgrading the master of a development cluster that many teams use to test their apps during the working day, a regional Google Kubernetes Engine (GKE) cluster in europe-west1. The upgrade ran through Jetstack's GKE Terraform pipeline and did not finish before the 20-minute Terraform timeout, while the GKE console still showed the cluster as upgrading.

   Upgrade stallControl plane unresponsiveNode recreation loopService outage

   Trigger

   A planned master upgrade of the shared development cluster. The same upgrade had been done on another pre-production environment earlier that day, and no API downtime was expected.

   Root cause

   Configuration change

   Component

   Admission webhookGKE control planeKubernetes API serverkubelet

   Contributing factor

   Excessive blast radiusNo health probesMonitoring blind spotAutomation amplification

   Detection

   The first sign was the master upgrade missing the 20-minute Terraform timeout; re-running the pipeline returned the kube-apiserver unhealthy error. The cause was identified with the help of Google Support.

   Remediation

   • Deleted the OPA ValidatingAdmissionWebhook resource during a window of API server access

   Action items

   • Limit the OPA webhook to the namespaces where the policy applies and development teams have access
   • Enable the webhook only for Ingress and Service resources, the only resources the policy validates
   • Add a liveness probe so OPA is restarted when it stops responding, and update the OPA documentation
   • Follow the open upstream Kubernetes issue about this behaviour

   Lessons

   With alerting on API server response times, the team might have seen CREATE and UPDATE requests slow down across the board as soon as the OPA-backed webhook was first deployed. Every workload needs probes: OPA looked so simple to deploy that the team skipped the Helm chart, which adds a livenessProbe for the admission controller containers among other adjustments to the tutorial configuration. Others had hit the same problem, and an upstream issue to improve it was still open.

   View case
4. IL-0076 Kubernetes masters went down after Dex filled etcd with AuthRequest objects and deleting its namespace started an endless list-and-timeout loop

   Fullstaq2019

   Symptoms

   Dex, the OIDC proxy that handled Kubernetes authentication, stored its data in the Kubernetes API (storage type kubernetes, inCluster: true) and had built up a huge number of AuthRequest objects. Listing all AuthRequests timed out, and without the list they could not be deleted.

   Control plane unresponsiveMonitoring outage

   Trigger

   The dex namespace was deleted so that Kubernetes would clean up the accumulated AuthRequest objects itself.

   Root cause

   Capacity exhaustion

   Component

   Kubernetes API serveretcdCluster authentication provider

   Contributing factor

   Unreviewed defaultsHarmful manual actionRetry stormLimited debug accessIgnored early signal

   Detection

   The speaker says he had got hints of the problem earlier. While the masters were down, plenty of alerts arrived in Slack, but monitoring could not be reached because authentication and port-forwarding no longer worked.

   Remediation

   • removed the kubernetes finalizer from the dex namespace by calling the API directly (kill-kube-ns)
   • deployed Dex to a new namespace with sane defaults

   Lessons

   Dex with Kubernetes as its storage keeps every AuthRequest as an API object, and with a long TTL they pile up until the API cannot list them. Deleting a namespace that holds that many objects makes Kubernetes list them in a loop and can overload the masters. Monitoring that is reached through cluster authentication is lost when authentication is what broke.

   View case
5. IL-0469 The managed Azure Kubernetes API server for the data100 hub stopped answering, so the stopped hub pod was not recreated

   UC Berkeley2019

   Symptoms

   On 25 February 2019 the Kubernetes API server of the managed Azure cluster running the data100 hub became unreachable. kubectl failed for every operation and requests to create new resources failed, so when the hub pod stopped, no new one was created and users saw a proxy error message.

   Control plane unresponsiveService outage

   Root cause

   Cloud provider outage

   Component

   AKS control planeKubernetes API serverkubelet

   Contributing factor

   Limited debug access

   Detection

   Infrastructure staff were notified in Slack at 11:34 and found that kubectl failed for all operations while the hub proxy was up.

   Remediation

   • Created a new cluster and requested a new NFS server in its resource group
   • Detached the data volumes from the old NFS server and moved them to the new cluster's resource group
   • Imported the ZFS pool into the new NFS server
   • Created the staging hub with helm and updated DNS
   • Created the prod hub with helm and updated DNS

   Action items

   • Monitor the API server endpoint remotely and alert when it is down

   View case

By root cause9