Cases6 cases

5 to 6 of 6

1. IL-0521 AKS pods could not mount volumes and new nodes got no CNI network interface after the cluster service principal expired

   Jenkins project2021

   Symptoms

   From 18:02 UTC on 2 June 2021, the Jenkins infrastructure AKS cluster logged errors about pods that were unable to mount volumes and about scaled-up nodes that were unable to obtain a network virtual interface from the CNI.

   Credentials unavailablePods stuck in ContainerCreating

   Trigger

   The Azure Service Principal credential of the AKS cluster expired at 17:53 UTC on 2 June 2021.

   Root cause

   Certificate or credential expiry

   Component

   Cluster cloud identityPersistent volumes and stateful setsCNI pluginNode pool

   Contributing factor

   Monitoring blind spot

   Detection

   The error behind the failures was found in the AKS UI console while the team investigated the root cause of the LTS 2.289.1 release problem.

   Remediation

   • Renewed the AKS cluster's Azure Service Principal credential, which made AKS restart all nodes automatically

   Action items

   • Add an automatic PagerDuty check that alerts a week before the service principal expires, as is done for certificates
   • Keep a team-level calendar for manual operations such as resetting a credential
   • Consider managed identities, which renew automatically, instead of the service principal

   View case
2. IL-0402 mybinder.org launches dropped to zero for hours because a JupyterHub pod on a lost GKE node kept its hub database PVC

   Binder Project2018

   Symptoms

   On 13 March 2018 at 07:28 CET the mybinder.org team noticed that JupyterHub had not been spawning new users and that the launch success rate on the Grafana dashboard had been at zero for several hours. Many pods were in the states "Unknown" and "NodeLost", on at least the nodes `10.128.0.7` and `10.128.0.5`.

   Service outageNode NotReadyPods stuck in TerminatingPods stuck in ContainerCreating

   Root cause

   unknown

   Component

   Persistent volumes and stateful setsNode pool

   Contributing factor

   Pages not delivered

   Detection

   The team noticed only after several hours, when the launch success rate on the Grafana dashboard had been at zero for a long time. No Stackdriver alert email arrived in time, because those emails were marked as spam and reached the team hours later.

   Remediation

   • Cordoned the bad nodes
   • Deleted the hub pod in Unknown state and the hub pod stuck in ContainerCreating
   • Reset node gke-prod-a-ssd-pool-32-134a959a-n2sk from the GCP web interface to force the PVC to be released
   • Force-deleted every pod in Unknown or NodeLost state with kubectl delete --grace-period=0 --force

   Action items

   • Investigate why the outage started, through Stackdriver logs and Google Cloud incidents
   • Set up an alerting service that does not depend on Stackdriver
   • Document how to check whether a PVC has not been released
   • Document how to restart a node by hand when commands are not working
   • Document how to delete pods that are in Unknown state

   Lessons

   The error went unnoticed for a long time because Stackdriver alert emails were marked as spam and arrived hours late. Once the problem was found, each step was logged and the steps were made more forceful as needed, so the hub was down only briefly after detection.

   View case

By root cause4