IL-0506 UC Berkeley · 2017-02-09kubernetesdatabase
UC Berkeley DataHub logins failed and students got empty home directories after the JupyterHub database was overwritten
Shortly after midnight on 9 February 2017, an operator debugging earlier 400 errors opened a shell in the DataHub hub pod, installed sqlite3, attached jupyterhub.sqlite and changed a user's server entry to see whether a wrong value would cause a problem. When nothing changed for the account, the operator started sqlite3 again without attaching the database and ran `.save jupyterhub.sqlite`, which overwrote the database file with an empty one; sqlite3 does this without any confirmation. All tables were gone. Users who logged in got new user ids, and because PVC names were built from the user id, the hub gave them new, empty disks.
At 1:28 AM the hub was scaled to 0 replicas and the team rebuilt the sqlite database from the Kubernetes PVCs, whose names contain a sanitized form of the username and the user id. The rebuilt database was attached at 2:42 AM and the admin page showed about 1000 users again. In the rebuild, cookie_id was left empty for many users, although JupyterHub expects it always to be a uuid, and everyone with an empty cookie_id was logged in as the first such user in the table. Changing one character near the end of the cookie secret had no effect, because JupyterHub uses only part of the secret; changing one at the beginning did log everyone out. In the meantime students were advised to use ds8.berkeley.edu, which crashed because it lacked resources for a data8-level class.
At 10:45 AM a script filled in cookie_id for all users, the hub was restarted and all user pods were deleted, and most users were back. Users with a '.' in their name had been rebuilt with a '-' instead, so those users attached to new empty PVCs; at 11:15 AM a script dropped the new users with '.' and renamed the older '-' users to use '.'. The all-clear was given at 11:17 AM.
- Severity
- –
- Scope
- UC Berkeley DataHub: logins failed with proxy errors, users saw empty home directories, and some were logged in as another user
- Users affected
- DataHub users, about 1000 accounts
- Region
- –
- Time to detect
- 35m
- Time to resolve
- 10h 52m
Symptoms
At about 1 AM on 9 February 2017, UC Berkeley's DataHub was reported down: users trying to log in got a proxy error. The hub pod was running, but its log was full of sqlite errors.
From 0:25:59 AM the hub logs showed "sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) no such table: proxies", and the same error followed for the hub table, the two tables used most often.
At 1:12 AM a student could log in but found all his notebooks gone. The JupyterHub admin panel listed only about 50 users where there had been about 1000, and `kubectl get pvc` showed that the student now had two PVCs, one 21 days old and one 5 minutes old: the hub had created a new account and a new, empty persistent disk because it had lost track of the old one.
After the database was rebuilt, some users were logged in as one particular other user. Changing the cookie secret and deleting all user pods logged everyone out but did not stop it. Later, users with a '.' in their username still got an empty home directory after everyone else was fine.
Root cause
Root cause Operator error
- Trigger
- While debugging, an operator ran `.save jupyterhub.sqlite` in sqlite3 inside the hub pod without having attached the database, which replaced the JupyterHub database file with an empty one.
Detection
DataHub was reported down at about 1 AM, when users got a proxy error at login. At 1:12 AM a user reported that he could log in but all his notebooks were gone.
Remediation
- Scaled the hub to 0 replicas
- Rebuilt the JupyterHub sqlite database by matching Kubernetes PVC names back to user ids
- Set cookie_id to null instead of a dummy value and attached the rebuilt database to the hub
- Changed the cookie secret in the hub pod and restarted it to log all users out
- Deleted all user pods to log everyone out
- Ran a script that filled in cookie_id for all users, restarted the hub and deleted all user pods again
- Dropped the new users with a '.' in their name and renamed the older users from '-' to '.'
Action items
- Report upstream that JupyterHub uses only part of the cookie secret and discards the rest
- Report upstream that the cookie_id column of the users table should have UNIQUE and NOT NULL constraints
- Ask KubeSpawner to support username hashes in PVC and pod names so the names stay the same when the database is deleted
- Ask OAuthenticator to support using the id from the Google authenticator as the user id, giving stable user ids
- Switch to Google Cloud SQL, a hosted and managed MySQL database
- Take regular and tested backups of the database
- Start an operational FAQ of things to do and not to do
- Set up better monitoring and paging
- Document escalation procedures explicitly
Lessons
–
Sources
References
Related cases
IL-0577Ficilcom's production Neon PostgreSQL database was reset by prisma migrate reset meant for the local databaseFicilcom · 2026same root cause · shared Database
IL-0576Every dataset in CareNet's production BigQuery data warehouse was deleted, and UNDROP SCHEMA could not restore themCareNet · 2025same root cause · shared Database
IL-0242Both PostgreSQL servers behind matrix.org were lost during a disk expansion, and the homeserver was down for 24 hoursMatrix.org Foundation · 2025same root cause · shared Database
IL-0095An engineer deleted the prod namespace of the LEAP JupyterHub while believing the kubeconfig pointed at a new cluster, and the hub was down for about an hour2i2c · 2025same root cause · shared Persistent volumes and stateful sets
IL-0582Jobs piled up in Kanmu's PostgreSQL-based qg job queue on Aurora as dequeue slowed and dead tuples kept growingKanmu · 2024same root cause · shared Database
IL-0556A transaction left open during database maintenance stopped uploads and then downloads on Storj's US1 satelliteStorj · 2024same root cause · shared Database
IL-0531Azure Database for MySQL Single Server and MariaDB in West US rejected every new connection for a batch of serversMicrosoft Azure · 2024same root cause · shared Database
IL-0565Cookie Run: Kingdom's CockroachDB cluster failed when 16 of 24 nodes lost their filesystems, causing a 36-hour outageDevsisters · 2022same root cause · shared Database
IL-0207A routine task run from an engineer's local environment switched to production and deleted GitLab.com's Consul release, so Patroni lost Consul and GitLab.com returned 5xx errors to all trafficGitLab · 2022same root cause · shared Database
IL-0197Moving a three-node Elasticsearch StatefulSet to new persistent volumes split the cluster in two, because the Helm chart hardcoded cluster.initial_master_nodes to elasticsearch-0Palark · 2021same root cause · shared Persistent volumes and stateful sets
IL-0332A test run against the production database truncated every table and travis-ci.com was down for about 5.5 hoursTravis CI · 2018same root cause · shared Database
IL-0431During Mackerel's move to AWS, PostgreSQL failed over to a replica that had stopped replicating, losing an hour of dataHatena · 2017same root cause · shared Database
IL-0218GitLab.com's primary PostgreSQL data was wiped by mistake and, with no working backup, the site was down about 18 hoursGitLab · 2017same root cause · shared Database
IL-0573A developer's mistake dropped nine main tables of the Baemin billing MariaDB database and payments briefly stoppedWoowa Brothers · 2016same root cause · shared Database
IL-0319Dropping databases on the MySQL replica mysql2 also dropped them on mysql1, as the pair actually ran master-masterOregon State University Open Source Lab · 2015same root cause · shared Database
IL-0337EBS volumes in one US East Availability Zone got stuck after a network change cut EBS nodes off from their replicasAmazon Web Services · 2011same root cause · shared Database
Check your cluster for this
For AI tools and scripts
- This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0506
- The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0506/similar
- MCP server (Streamable HTTP, no key):
https://incidentlib-mcp.benchlet.com/mcp
The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.