Incident Library
Applied filters SymptomData lossFactorNo safeguard on destructive changeClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases8 cases

8 to 8 of 8

  1. IL-0506 UC Berkeley DataHub logins failed and students got empty home directories after the JupyterHub database was overwritten
    UC Berkeley2017
    Symptoms

    At about 1 AM on 9 February 2017, UC Berkeley's DataHub was reported down: users trying to log in got a proxy error. The hub pod was running, but its log was full of sqlite errors.

    Service outageData lossMisrouted requests
    Trigger

    While debugging, an operator ran `.save jupyterhub.sqlite` in sqlite3 inside the hub pod without having attached the database, which replaced the JupyterHub database file with an empty one.

    Root cause
    Operator error
    Component
    DatabasePersistent volumes and stateful setsJupyterHub
    Contributing factor
    No safeguard on destructive changeUnsafe failure behaviourHarmful manual action
    Detection

    DataHub was reported down at about 1 AM, when users got a proxy error at login. At 1:12 AM a user reported that he could log in but all his notebooks were gone.

    Remediation
    • Scaled the hub to 0 replicas
    • Rebuilt the JupyterHub sqlite database by matching Kubernetes PVC names back to user ids
    • Set cookie_id to null instead of a dummy value and attached the rebuilt database to the hub
    • Changed the cookie secret in the hub pod and restarted it to log all users out
    • Deleted all user pods to log everyone out
    • Ran a script that filled in cookie_id for all users, restarted the hub and deleted all user pods again
    • Dropped the new users with a '.' in their name and renamed the older users from '-' to '.'
    Action items
    • Report upstream that JupyterHub uses only part of the cookie secret and discards the rest
    • Report upstream that the cookie_id column of the users table should have UNIQUE and NOT NULL constraints
    • Ask KubeSpawner to support username hashes in PVC and pod names so the names stay the same when the database is deleted
    • Ask OAuthenticator to support using the id from the Google authenticator as the user id, giving stable user ids
    • Switch to Google Cloud SQL, a hosted and managed MySQL database
    • Take regular and tested backups of the database
    • Start an operational FAQ of things to do and not to do
    • Set up better monitoring and paging
    • Document escalation procedures explicitly
    View case
Check your cluster for this

These 8 cases involve Database and Database replication. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause2

Operator error7 cases

Component
Database6Database backups and snapshots1Database replication1JupyterHub1Message broker1Persistent volumes and stateful sets1
Contributing factor
No safeguard on destructive change7Harmful manual action4Overlapping failures2Signal buried in noise1Unpatched known bug1Retry storm1Unrepresentative staging1Unsafe failure behaviour1
Organisation
CareNet1Devsisters1Ficilcom1Honeycomb1Matrix.org Foundation1Travis CI1UC Berkeley1
  1. IL-0577 Ficilcom's production Neon PostgreSQL database was reset by prisma migrate reset meant for the local database
    Ficilcom2026
  2. IL-0286 An exercise cleanup destroyed Honeycomb's Kafka brokers, leaving partitions leaderless and ingest crash-looping
    Honeycomb2025
  3. IL-0576 Every dataset in CareNet's production BigQuery data warehouse was deleted, and UNDROP SCHEMA could not restore them
    CareNet2025
  4. IL-0242 Both PostgreSQL servers behind matrix.org were lost during a disk expansion, and the homeserver was down for 24 hours
    Matrix.org Foundation2025
  5. IL-0565 Cookie Run: Kingdom's CockroachDB cluster failed when 16 of 24 nodes lost their filesystems, causing a 36-hour outage
    Devsisters2022
  6. IL-0332 A test run against the production database truncated every table and travis-ci.com was down for about 5.5 hours
    Travis CI2018
  7. IL-0506 UC Berkeley DataHub logins failed and students got empty home directories after the JupyterHub database was overwritten
    UC Berkeley2017

Certificate or credential expiry1 case

Component
DatabaseDatabase replication
Contributing factor
Unverified assumptionChange with a delayed effectMonitoring blind spotNo safeguard on destructive change
Organisation
LootLocker
  1. IL-0490 Records were missing from LootLocker's new MySQL database after a migration cutover and the service degraded for 3 hours
    LootLocker2026