Incident Library

Component

JupyterHub

JupyterHub · jupyterhub

4 cases · 2017 to 2022

Root cause Capacity exhaustion2Operator error1Software defect1
Contributing factor No alert on limit3Harmful manual action2No resource requests or limits1No safeguard on destructive change1Unsafe failure behaviour1
Symptom Service outage3Container crash loop1Data loss1High error rate1OOMKilled1Pods Pending1Probe failures1Misrouted requests1Stale or inconsistent API reads1Persistent volume full1
Organisation Binder Project22i2c1UC Berkeley1

Cases, by severity

  1. IL-0500 UToronto JupyterHub logins and server starts returned 500 errors when the hub's sqlite database disk filled up2i2c · 2022

    Root cause Capacity exhaustion

  2. IL-0504 mybinder.org launches failed for about five hours as undeleted user pods piled up and filled the clusterBinder Project · 2018

    Root cause Software defect

  3. IL-0503 mybinder.org launches failed after a GKE node ran out of memory and its nginx-ingress proxy pod crash-loopedBinder Project · 2017

    Root cause Capacity exhaustion

  4. IL-0506 UC Berkeley DataHub logins failed and students got empty home directories after the JupyterHub database was overwrittenUC Berkeley · 2017

    Root cause Operator error

View on the graph