Cases6 cases

6 to 6 of 6

1. IL-0507 Student pods on UC Berkeley DataHub got stuck in Terminating or would not start because Docker hung on two Azure nodes

   UC Berkeley2017

   Symptoms

   On 10 October 2017 at 10:45 AM, a post on the course Piazza reported that two students could not start their servers on UC Berkeley's DataHub, and the /hub/admin interface could not start them either. At first it was thought the students might have run out of memory.

   Pods stuck in Terminating

   Root cause

   unknown

   Component

   Container runtime

   Contributing factor

   –

   Detection

   Students reported on the course Piazza at 10:45 AM, and later in the course Slack, that their servers would not start.

   Remediation

   • Force-deleted the stuck user pods with kubectl --namespace=prod delete pod jupyter-<name> --grace-period=0 --force
   • Cordoned node k8s-pool1-19522833-13
   • Force-deleted the stuck pods on k8s-pool2-19522833-9 and cordoned that node
   • Stopped and started k8s-pool1-19522833-13 from the Azure portal, after which Docker responded, and uncordoned it
   • Stopped and started k8s-pool2-19522833-9 from the Azure portal

   Action items

   • When several student servers fail to start or stop, check whether their pods ran on the same nodes
   • Find out how many nodes do not mount /var/lib/docker on sdb1
   • Watch for elevated counts of pods stuck in Terminating

   View case

By root cause5