Incident Library
Applied filters ComponentJupyterHubSymptomHigh error rateFactorNo alert on limitClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0500 UToronto JupyterHub logins and server starts returned 500 errors when the hub's sqlite database disk filled up
    2i2c2022
    Symptoms

    On 31 October 2022 at 6:27 PM Pacific time, users of the UToronto JupyterHub run by 2i2c could no longer log in or start their servers, and the hub answered with 500 errors. Users who were already logged in could keep using the hub.

    High error ratePersistent volume full
    Trigger

    The JupyterHub log file, written to the same 10G disk as the hub's sqlite database, grew until the disk was completely full.

    Root cause
    Capacity exhaustion
    Component
    DatabasePersistent volumes and stateful setsJupyterHub
    Contributing factor
    No alert on limit
    Detection

    A user emailed 2i2c support at 7:13 PM that users could not log in; the team acknowledged the issue at 8:33 PM. There was no alert for the problem.

    Remediation
    • Compressed the hub log file into /tmp from a shell in the hub pod, because it could not be compressed in place on the full disk
    • Copied the compressed log to a local machine so the hub pod could be restarted without losing it
    • Resized the hub database disk (the hub-db-dir PVC) from 10G to 60G
    • Restarted the hub pod, which came back with the larger disk and restored service
    Action items
    • Stop keeping hub logs on the same disk as the hub database
    • Add a Grafana graph of free space on the JupyterHub database disk
    • Alert on elevated error rates
    Lessons

    Kubernetes dynamic volume resizing let the hub database disk grow from 10G to 60G without much effort. The risk of keeping logs on the database disk had been identified during the earlier migration, but the team had no solid process for following it through, and because no alert existed the team had to wait for a person to report the outage.

    View case
Check your cluster for this

This case involves Database and JupyterHub. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Capacity exhaustion1 case

Component
DatabaseJupyterHubPersistent volumes and stateful sets
Contributing factor
No alert on limit
Organisation
2i2c
  1. IL-0500 UToronto JupyterHub logins and server starts returned 500 errors when the hub's sqlite database disk filled up
    2i2c2022