Cases1 case

1 to 1 of 1

1. IL-0468 New JupyterHub student pods stuck Pending on MaxVolumeCount after a memory limit cut with node size unchanged

   UC Berkeley2017

   Symptoms

   On 20 March 2017, from some time early in the day until about 13:00, some new student servers on UC Berkeley's JupyterHub stayed in Pending forever and the students got 500 errors that kept them from logging in.

   Pods PendingHigh error rate

   Trigger

   On 18 March at 16:30 the RAM per student was reduced from 2G to 1G to save resources while the nodes kept the same size of 26G RAM, and as a result the nodes hit the Google Cloud the limit on the number of disks per node.

   Root cause

   Capacity exhaustion

   Component

   Container memory limitCloud block storageNode pool

   Contributing factor

   Unknown limit

   Detection

   New student servers reported a 500 error that kept students from logging in. At 12:44 this was judged widespread, and at 12:53 `kubectl describe pod` showed the pods Pending on MaxVolumeCount.

   Remediation

   • Copied the Google Container Engine instance template and changed it to spawn smaller n1-highmem-2 nodes
   • Switched the managed instance group to the new template
   • Grew the instance group by 30 new nodes
   • Ran populate.bash so student pods would start on time on the new nodes
   • Stopped the simple autoscaler
   • Stopped and started affected user pods from the control panel
   • Restarted the hub and proxy pods

   Action items

   • Follow the status of kubernetes/kubernetes issue 24317
   • Find and fix the cause of the 'too many redirects' error in JupyterHub
   • Delete the older, larger nodes that are no longer in use
   • Alert when any pods stay Pending for a non-negligible time

   View case

By root cause1