Cases3 cases

1 to 3 of 3

1. IL-0500 UToronto JupyterHub logins and server starts returned 500 errors when the hub's sqlite database disk filled up

   2i2c2022

   Symptoms

   On 31 October 2022 at 6:27 PM Pacific time, users of the UToronto JupyterHub run by 2i2c could no longer log in or start their servers, and the hub answered with 500 errors. Users who were already logged in could keep using the hub.

   High error ratePersistent volume full

   Trigger

   The JupyterHub log file, written to the same 10G disk as the hub's sqlite database, grew until the disk was completely full.

   Root cause

   Capacity exhaustion

   Component

   DatabasePersistent volumes and stateful setsJupyterHub

   Contributing factor

   No alert on limit

   Detection

   A user emailed 2i2c support at 7:13 PM that users could not log in; the team acknowledged the issue at 8:33 PM. There was no alert for the problem.

   Remediation

   • Compressed the hub log file into /tmp from a shell in the hub pod, because it could not be compressed in place on the full disk
   • Copied the compressed log to a local machine so the hub pod could be restarted without losing it
   • Resized the hub database disk (the hub-db-dir PVC) from 10G to 60G
   • Restarted the hub pod, which came back with the larger disk and restored service

   Action items

   • Stop keeping hub logs on the same disk as the hub database
   • Add a Grafana graph of free space on the JupyterHub database disk
   • Alert on elevated error rates

   Lessons

   Kubernetes dynamic volume resizing let the hub database disk grow from 10G to 60G without much effort. The risk of keeping logs on the database disk had been identified during the earlier migration, but the team had no solid process for following it through, and because no alert existed the team had to wait for a person to report the outage.

   View case
2. IL-0504 mybinder.org launches failed for about five hours as undeleted user pods piled up and filled the cluster

   Binder Project2018

   Symptoms

   On Saturday 8 July 2018 at about 18:30 CEST, the launch success rate of mybinder.org dropped to zero and stayed there. Suspicious behaviour was first reported on Gitter at 22:48, more than four hours later.

   Service outagePods PendingStale or inconsistent API reads

   Root cause

   Software defect

   Component

   JupyterHub

   Contributing factor

   No alert on limit

   Detection

   Suspicious behaviour was reported on Gitter at 22:48 CEST, more than four hours after launch success dropped to zero; no downtime notifications were set up.

   Remediation

   • Deleted user pods older than four hours with the delete-pods.py script
   • Culled pods of the most-used images, such as ipython-in-depth, more aggressively
   • Restarted the Hub pod after about 600 pods were culled, which brought launches back to normal
   • Cordoned some of the extra nodes so the autoscaler could reclaim them
   • Ran manual max-age culling the next morning, letting the cluster scale back to 3 nodes

   Action items

   • Set up notifications of downtime
   • Automate cluster scale-down
   • Document the downtime recovery steps so every team member knows them
   • Fix KubeSpawner's failure to delete pods when the pod reflector fails
   • Make KubeSpawner more robust to pod reflector failure
   • Refactor the event reflector into a single reflector instead of one per pod
   • Detect a likely unhealthy Hub state and restart the Hub without manual intervention

   Lessons

   The delete-pods.py script was useful for bulk-deleting old pods, and once the problem was noticed the cluster was healthy within a few minutes. The outage lasted five hours before anyone noticed because no downtime notifications existed, and scaling the cluster back down still took several manual steps of cordoning and draining nodes.

   View case
3. IL-0503 mybinder.org launches failed after a GKE node ran out of memory and its nginx-ingress proxy pod crash-looped

   Binder Project2017

   Symptoms

   On 30 November 2017 at 16:59 PST, mybinder.org operators noticed that Grafana looked wrong and that launching a new server failed with an error that a temporary user could not be created. Pods that were already running kept running, but no new ones were being created.

   Service outageContainer crash loopProbe failuresOOMKilled

   Trigger

   A build container, which ran without a memory limit, most likely used up all the memory on the node that ran the proxy pod.

   Root cause

   Capacity exhaustion

   Component

   Ingress controllerContainer memory limitJupyterHub

   Contributing factor

   No resource requests or limitsHarmful manual actionNo alert on limit

   Detection

   An operator happened to look at Grafana at 16:59 PST, saw that it looked wrong, and confirmed the error by launching a new server.

   Remediation

   • Cordoned the node that had run out of memory
   • Uncordoned the node once the proxy was working

   Action items

   • Limit the memory available to build pods
   • Change the JupyterHub configuration so user pods are not cleaned up when the jupyterhub pod restarts
   • Nominate one person to drive the response, possibly in a shared terminal, so only one person changes the cluster
   • Alert when something is wrong with the cluster, for example by counting 4xx responses on the jupyterhub pod

   View case

By root cause2