Cases1 case
1 to 1 of 1
IL-0500UToronto JupyterHub logins and server starts returned 500 errors when the hub's sqlite database disk filled up2i2c2022- Symptoms
On 31 October 2022 at 6:27 PM Pacific time, users of the UToronto JupyterHub run by 2i2c could no longer log in or start their servers, and the hub answered with 500 errors. Users who were already logged in could keep using the hub.
- Trigger
The JupyterHub log file, written to the same 10G disk as the hub's sqlite database, grew until the disk was completely full.
- Root cause
- Component
- Contributing factor
- Detection
A user emailed 2i2c support at 7:13 PM that users could not log in; the team acknowledged the issue at 8:33 PM. There was no alert for the problem.
- Remediation
- Compressed the hub log file into /tmp from a shell in the hub pod, because it could not be compressed in place on the full disk
- Copied the compressed log to a local machine so the hub pod could be restarted without losing it
- Resized the hub database disk (the hub-db-dir PVC) from 10G to 60G
- Restarted the hub pod, which came back with the larger disk and restored service
- Action items
- Stop keeping hub logs on the same disk as the hub database
- Add a Grafana graph of free space on the JupyterHub database disk
- Alert on elevated error rates
- Lessons
Kubernetes dynamic volume resizing let the hub database disk grow from 10G to 60G without much effort. The risk of keeping logs on the database disk had been identified during the earlier migration, but the team had no solid process for following it through, and because no alert existed the team had to wait for a person to report the outage.
This case involves Database and JupyterHub. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.