Incident Library
Applied filters Root causeCapacity exhaustionComponentJupyterHubClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0500 UToronto JupyterHub logins and server starts returned 500 errors when the hub's sqlite database disk filled up
    2i2c2022
    Symptoms

    On 31 October 2022 at 6:27 PM Pacific time, users of the UToronto JupyterHub run by 2i2c could no longer log in or start their servers, and the hub answered with 500 errors. Users who were already logged in could keep using the hub.

    High error ratePersistent volume full
    Trigger

    The JupyterHub log file, written to the same 10G disk as the hub's sqlite database, grew until the disk was completely full.

    Root cause
    Capacity exhaustion
    Component
    DatabasePersistent volumes and stateful setsJupyterHub
    Contributing factor
    No alert on limit
    Detection

    A user emailed 2i2c support at 7:13 PM that users could not log in; the team acknowledged the issue at 8:33 PM. There was no alert for the problem.

    Remediation
    • Compressed the hub log file into /tmp from a shell in the hub pod, because it could not be compressed in place on the full disk
    • Copied the compressed log to a local machine so the hub pod could be restarted without losing it
    • Resized the hub database disk (the hub-db-dir PVC) from 10G to 60G
    • Restarted the hub pod, which came back with the larger disk and restored service
    Action items
    • Stop keeping hub logs on the same disk as the hub database
    • Add a Grafana graph of free space on the JupyterHub database disk
    • Alert on elevated error rates
    Lessons

    Kubernetes dynamic volume resizing let the hub database disk grow from 10G to 60G without much effort. The risk of keeping logs on the database disk had been identified during the earlier migration, but the team had no solid process for following it through, and because no alert existed the team had to wait for a person to report the outage.

    View case
  2. IL-0503 mybinder.org launches failed after a GKE node ran out of memory and its nginx-ingress proxy pod crash-looped
    Binder Project2017
    Symptoms

    On 30 November 2017 at 16:59 PST, mybinder.org operators noticed that Grafana looked wrong and that launching a new server failed with an error that a temporary user could not be created. Pods that were already running kept running, but no new ones were being created.

    Service outageContainer crash loopProbe failuresOOMKilled
    Trigger

    A build container, which ran without a memory limit, most likely used up all the memory on the node that ran the proxy pod.

    Root cause
    Capacity exhaustion
    Component
    Ingress controllerContainer memory limitJupyterHub
    Contributing factor
    No resource requests or limitsHarmful manual actionNo alert on limit
    Detection

    An operator happened to look at Grafana at 16:59 PST, saw that it looked wrong, and confirmed the error by launching a new server.

    Remediation
    • Cordoned the node that had run out of memory
    • Uncordoned the node once the proxy was working
    Action items
    • Limit the memory available to build pods
    • Change the JupyterHub configuration so user pods are not cleaned up when the jupyterhub pod restarts
    • Nominate one person to drive the response, possibly in a shared terminal, so only one person changes the cluster
    • Alert when something is wrong with the cluster, for example by counting 4xx responses on the jupyterhub pod
    View case
Check your cluster for this

These 2 cases involve JupyterHub and Database. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Capacity exhaustion2 cases

Component
JupyterHub2Database1Ingress controller1Container memory limit1Persistent volumes and stateful sets1
Contributing factor
No alert on limit2Harmful manual action1No resource requests or limits1
Organisation
2i2c1Binder Project1
  1. IL-0500 UToronto JupyterHub logins and server starts returned 500 errors when the hub's sqlite database disk filled up
    2i2c2022
  2. IL-0503 mybinder.org launches failed after a GKE node ran out of memory and its nginx-ingress proxy pod crash-looped
    Binder Project2017