IL-0500 2i2c · 2022-10-31kubernetesdatabase
UToronto JupyterHub logins and server starts returned 500 errors when the hub's sqlite database disk filled up
The UToronto JupyterHub stored its hub logs on the same 10G disk as the hub's sqlite database. It was the only hub run by 2i2c with this layout. The commit that set it up during an earlier migration noted that the layout was not ideal and only copied the hub's prior configuration; the write-up says the likely reason was that the hub did not use Azure for logging at all and the team wanted to keep detailed logs for analytics. The risk had been identified during that migration, but there was no process to follow it through. The log file eventually filled the disk, database operations could no longer succeed, and hub logins and server starts failed from 6:27 PM until about 9:00 PM Pacific time, 2 hours 33 minutes in all.
Once the issue was acknowledged, the hub logs made the cause clear. The log file could not be compressed in place on the full disk, so an engineer opened a shell in the hub pod with kubectl exec and gzipped jupyterhub.log into /tmp, which cut its size by about ten times; the log was kept rather than deleted in case it was needed for analytics. Deleting the uncompressed file would not free space while the JupyterHub process still held it open, so the hub process had to restart, and because a pod restart would also lose /tmp, the compressed file was first copied to a local machine with kubectl cp. While the copy ran, the hub-db-dir PVC was edited from 10G to 60G, so that a single pod restart applied the resize too. The hub pod restarted at 8:59 PM with the larger disk and service came back. The compressed log was copied back to the hub disk at 10:01 PM, and the hub kept writing its logs to that disk after the incident.
- Severity
- –
- Scope
- UToronto JupyterHub: new logins and server starts failed with 500 errors; users already logged in kept working
- Users affected
- UToronto JupyterHub users who were not already logged in
- Region
- –
- Time to detect
- 46m
- Time to resolve
- 2h 33m
Symptoms
On 31 October 2022 at 6:27 PM Pacific time, users of the UToronto JupyterHub run by 2i2c could no longer log in or start their servers, and the hub answered with 500 errors. Users who were already logged in could keep using the hub.
The hub pod's logs contained "Rolling back session due to database error" from JupyterHub, followed by a SQLAlchemy traceback that ended in "sqlite3.OperationalError: database or disk is full".
Nothing alerted on the problem. A user emailed 2i2c support at 7:13 PM, the team acknowledged the issue at 8:33 PM, and at 8:47 PM the hub logs and the hub configuration showed that the hub's log file and its sqlite database shared one disk, and that the disk was full. The start time was worked out later from Grafana logs of the last successful login.
Root cause
Root cause Capacity exhaustion
- Trigger
- The JupyterHub log file, written to the same 10G disk as the hub's sqlite database, grew until the disk was completely full.
Detection
A user emailed 2i2c support at 7:13 PM that users could not log in; the team acknowledged the issue at 8:33 PM. There was no alert for the problem.
Remediation
- Compressed the hub log file into /tmp from a shell in the hub pod, because it could not be compressed in place on the full disk
- Copied the compressed log to a local machine so the hub pod could be restarted without losing it
- Resized the hub database disk (the hub-db-dir PVC) from 10G to 60G
- Restarted the hub pod, which came back with the larger disk and restored service
Action items
- Stop keeping hub logs on the same disk as the hub database
- Add a Grafana graph of free space on the JupyterHub database disk
- Alert on elevated error rates
Lessons
Kubernetes dynamic volume resizing let the hub database disk grow from 10G to 60G without much effort. The risk of keeping logs on the database disk had been identified during the earlier migration, but the team had no solid process for following it through, and because no alert existed the team had to wait for a person to report the outage.
Sources
References
- https://github.com/2i2c-org/infrastructure/pull/886/commits/9c30396a46202de639f12f5b924cd9fe03a8f086
- https://github.com/2i2c-org/infrastructure/blob/1f916f8851fa636af065d427ba70267c55c8c1de/config/clusters/utoronto/common.values.yaml#L89
- https://kubernetes.io/blog/2018/07/12/resizing-persistent-volumes-using-kubernetes/
- https://github.com/2i2c-org/infrastructure/pull/1847
- https://github.com/jupyterhub/grafana-dashboards/pull/49
- https://github.com/2i2c-org/infrastructure/issues/1804
Related cases
IL-0450Oracle RAC commits behind order APIs kept waiting over 0.1 s on log file sync while average host CPU had headroomToss Securities · 2026same root cause · shared Database
IL-0553CircleCI's orchestration database filled its transaction log volume, and no workflows or jobs started for 50 minutesCircleCI · 2026same root cause · shared Database
IL-0379InfluxDB Cloud queries in Azure West Europe slowed and timed out as storage pods fell behind and kept restartingInfluxData · 2026same root cause · shared Database
IL-0309Preset's RDS PostgreSQL 14.7 to 17.9 upgrade in US East 1 ran out of storage and took Superset down for six hoursPreset · 2026same root cause · shared Database
IL-0435Error rates rose as Aurora MySQL killed connections and declined queries to avoid OOM while load metrics stayed flatCyberAgent · 2026same root cause · shared Database
IL-0497Buttondown's database ran out of connections twice in one day and requests failed while health checks mostly passedButtondown · 2026same root cause · shared Database
IL-0288ZooKeeperWait rose to thousands of seconds on three redundant ClickHouse instances, degrading a CDN77 data pipelineCDN77 · 2026same root cause · shared Database
IL-0350Hosts in an IONOS TXL cluster lost connectivity when an InfiniBand fabric resweep failed on a full multicast tableIONOS Cloud · 2026same root cause · shared Database
IL-0305Replicas of one Intercom Vitess shard hit 100% CPU and fell up to 2.2 hours behind, serving stale inbox dataIntercom · 2026same root cause · shared Database
IL-0390PostHog US Cloud event ingestion slowed to delays of up to 2 days as writes to the Persons Postgres table stalledPostHog · 2025same root cause · shared Database
IL-0100PostHog's feature flags pods were packed onto Kubernetes nodes that were too small for their CPU requests, and CPU saturation above 90% caused three outages in ten days through Redis overload, connection pool exhaustion and slow queriesPostHog · 2025same root cause · shared Database
IL-0287One overloaded shard of MongoDB's internal monitoring database delayed Atlas metrics, alerts and cluster operationsMongoDB · 2025same root cause · shared Database
IL-0557Database transaction locks piled up on Storj's US1 satellite, and uploads and downloads failed with timeouts for 2 hoursStorj · 2025same root cause · shared Database
IL-0067Redis filled its persistent volume, corrupted its append-only file and crash-looped on every restartBC Government · 2025 · sev1same root cause · shared Persistent volumes and stateful sets
IL-0551Bitbucket Cloud's overloaded primary database failed over, and the website, API and Pipelines were slow for almost a dayAtlassian · 2025same root cause · shared Database
IL-0085A control plane node of Magic Pages' Kubernetes cluster on Hetzner Cloud ran out of memory, the cluster's own calls then used up the Hetzner API rate limit, and all customer websites were down for about 7.5 hoursMagic Pages · 2025same root cause · shared Persistent volumes and stateful sets
IL-0298Wikimedia edits failed for 38 minutes when sessionstore Cassandra nodes in both data centres crashed on full disksWikimedia Foundation · 2025same root cause · shared Database
IL-0312Gadget's main Postgres did not start again after a restart to raise max_connections, and the platform went downGadget · 2024same root cause · shared Database
IL-0544Wasabi's database cache stopped responding when its queueing service filled up, and S3 failed in all regionsWasabi · 2024same root cause · shared Database
IL-0447An every RDS for MySQL instance slowed badly when its gp2 EBS BurstBalance ran out and IOPS fell to the baselineevery · 2024same root cause · shared Database
IL-0375MongoDB Atlas cluster changes and alerts lagged for hours after a control plane metadata migration congested a databaseMongoDB · 2024same root cause · shared Database
IL-0110A Grafana Agent pod's high memory use and crash-looping Hangfire pods drove a Kubernetes node into system OOM and NotReady, twice in one dayDFDS · 2024same root cause · shared Database
IL-0550Bitbucket Cloud's database slowed until connection pools saturated and requests timed out, leaving Pipelines pendingAtlassian · 2024same root cause · shared Database
IL-0270Discord's ScyllaDB authentication cluster overloaded during an OS upgrade, and logins failed for about 50 minutesDiscord · 2023same root cause · shared Database
IL-0202A burst of queries and database writes exhausted connection pools and memory on Honeycomb's ingest hosts, and Kubernetes crash-loop back-off kept failed hosts offline so ingest capacity stayed too low to recoverHoneycomb · 2023 · sev1same root cause · shared Database
IL-0386After a move to Google Cloud, Postgres connection timeouts and memcached i/o timeouts spiked on GKE nodesincident.io · 2023same root cause · shared Database
IL-0326A retention clean-up job filled the disk of MailPace's managed Postgres, which stayed offline for just over two hoursMailPace · 2023same root cause · shared Database
IL-0546Buildkite jobs could not go to newly connected agents once agent IDs passed a 32-bit integer foreign key's limitBuildkite · 2023same root cause · shared Database
IL-0301Basecamp 3's primary database went into total gridlock at the top of the hour and Basecamp 3 went down twice37signals · 2022same root cause · shared Database
IL-0206Consul agents in GitLab.com's GKE pods were repeatedly OOM-killed, so the Rails pods lost database replica discovery and sent every query to the primaryGitLab · 2022same root cause · shared Database
IL-0302Peak load on GitHub's mysql1 MySQL cluster maxed out its database proxy's connections and stopped writes in March 2022GitHub · 2022same root cause · shared Database
IL-0212Many Slack users could not connect because a Vitess keyspace for channel membership was overloaded by cache-miss queriesSlack · 2022same root cause · shared Database
IL-0456Inflearn went down six times during a January 2022 free-course event as slow queries piled up on Aurora PostgreSQLInflab · 2022same root cause · shared Database
IL-0321GCP snapshots on all Patroni nodes stalled GitLab.com's primary PostgreSQL disk, Patroni failed over and 5xx spikedGitLab · 2021same root cause · shared Database
IL-0496Ably's Cassandra hit 100% CPU in every region and queries timed out globally, and the load returned on two more eveningsAbly · 2020same root cause · shared Database
IL-0443A Repro Aurora MySQL reader restarted unexpectedly and writer insert latency rose about sevenfold for a day or moreRepro · 2019same root cause · shared Database
IL-0037Image registry rate-limited a whole region after a crash-looping DaemonSet with imagePullPolicy Always pulled through three NAT instancesDatadog · 2019same root cause · shared Database
IL-0297Mezmo users could not sign in and log indexing lagged after the auth cache's Redis servers were OOM killedMezmo · 2019same root cause · shared Database
IL-0211A Mandrill PostgreSQL shard shut down at transaction ID wraparound and email sending was impaired for about 40 hoursMailchimp · 2019same root cause · shared Database
IL-0064CPU-heavy pods packed onto one Google Cloud node drove it to kernel panics, and the crash moved from node to node until the site was downMoonlight · 2019same root cause · shared Database
IL-0300Basecamp 3 went read-only for almost five hours when its events table reached the ceiling of 2,147,483,64737signals · 2018same root cause · shared Database
IL-0518gitlab.freedesktop.org on GKE ran short of volume space and a planned 30-minute storage upgrade became a 4-hour outagefreedesktop.org · 2018same root cause · shared Persistent volumes and stateful sets
IL-0503mybinder.org launches failed after a GKE node ran out of memory and its nginx-ingress proxy pod crash-loopedBinder Project · 2017same root cause · shared JupyterHub
IL-0334Instapaper's RDS MySQL bookmarks table hit the 2TB ext3 file size limit, inserts failed and the service went downInstapaper · 2017same root cause · shared Database
IL-0333A downsized RDS PostgreSQL database maxed out its CPU at US peak and failing ELB health checks took buildkite.com downBuildkite · 2016same root cause · shared Database
IL-0282DynamoDB in US-East reached about 55% request errors as storage servers overloaded its metadata serviceAmazon Web Services · 2015same root cause · shared Database
IL-0383PagerDuty's shared Cassandra cluster degraded during a repair, delaying notifications and losing incoming eventsPagerDuty · 2014 · sev1same root cause · shared Database
IL-0320Postmark's primary MySQL disk filled but MySQL kept running, so MySQL-MMM never failed over and sending was delayedPostmark · 2011same root cause · shared Database
Check your cluster for this
For AI tools and scripts
- This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0500
- The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0500/similar
- MCP server (Streamable HTTP, no key):
https://incidentlib-mcp.benchlet.com/mcp
The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.