Incident Library
Applied filters ComponentPersistent volumes and stateful setsDomainKubernetesClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases16 cases

16 to 16 of 16

  1. IL-0506 UC Berkeley DataHub logins failed and students got empty home directories after the JupyterHub database was overwritten
    UC Berkeley2017
    Symptoms

    At about 1 AM on 9 February 2017, UC Berkeley's DataHub was reported down: users trying to log in got a proxy error. The hub pod was running, but its log was full of sqlite errors.

    Service outageData lossMisrouted requests
    Trigger

    While debugging, an operator ran `.save jupyterhub.sqlite` in sqlite3 inside the hub pod without having attached the database, which replaced the JupyterHub database file with an empty one.

    Root cause
    Operator error
    Component
    DatabasePersistent volumes and stateful setsJupyterHub
    Contributing factor
    No safeguard on destructive changeUnsafe failure behaviourHarmful manual action
    Detection

    DataHub was reported down at about 1 AM, when users got a proxy error at login. At 1:12 AM a user reported that he could log in but all his notebooks were gone.

    Remediation
    • Scaled the hub to 0 replicas
    • Rebuilt the JupyterHub sqlite database by matching Kubernetes PVC names back to user ids
    • Set cookie_id to null instead of a dummy value and attached the rebuilt database to the hub
    • Changed the cookie secret in the hub pod and restarted it to log all users out
    • Deleted all user pods to log everyone out
    • Ran a script that filled in cookie_id for all users, restarted the hub and deleted all user pods again
    • Dropped the new users with a '.' in their name and renamed the older users from '-' to '.'
    Action items
    • Report upstream that JupyterHub uses only part of the cookie secret and discards the rest
    • Report upstream that the cookie_id column of the users table should have UNIQUE and NOT NULL constraints
    • Ask KubeSpawner to support username hashes in PVC and pod names so the names stay the same when the database is deleted
    • Ask OAuthenticator to support using the id from the Google authenticator as the user id, giving stable user ids
    • Switch to Google Cloud SQL, a hosted and managed MySQL database
    • Take regular and tested backups of the database
    • Start an operational FAQ of things to do and not to do
    • Set up better monitoring and paging
    • Document escalation procedures explicitly
    View case
Check your cluster for this

These 16 cases involve Persistent volumes and stateful sets and Database. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause9

Configuration change2 cases

Component
Persistent volumes and stateful sets2Argo CD1In-memory cache1Configuration templating1Container memory limit1
Contributing factor
Unverified assumption1Automation amplification1Monitoring blind spot1No health probes1No runbook1No safeguard on destructive change1No headroom1Retry storm1Missed in review1Single point of failure1
Organisation
AdGuard VPN1InfluxData1
  1. IL-0063 VPN outage after recreating a Redis StatefulSet restarted the whole Redis cluster without readiness checks
    AdGuard VPN2025
  2. IL-0009 A duplicated generated name made Argo CD replace the InfluxDB Cloud production Application and delete the whole production environment
    InfluxData2022

Breaking change in an upgrade1 case

Component
ConfigMaps and SecretsClient-side DNS resolver and cacheGKE control planeNode poolPersistent volumes and stateful sets
Contributing factor
No headroomIneffective rollbackSingle point of failureDeferred upgrades
Organisation
Wealthy
  1. IL-0101 After Google forced a GKE control plane upgrade to 1.33.2 at a deprecation deadline, no new node could initialise, so autoscaling and spot node replacement stopped for 15 hours
    Wealthy2025

Operator error3 cases

Component
Persistent volumes and stateful sets3Database1Elasticsearch or OpenSearch1Helm1Ingress controller1Jobs and CronJobs1JupyterHub1Terraform1
Contributing factor
No safeguard on destructive change2Unverified assumption1Harmful manual action1No runbook1Unsafe failure behaviour1
Organisation
2i2c1Palark1UC Berkeley1
  1. IL-0095 An engineer deleted the prod namespace of the LEAP JupyterHub while believing the kubeconfig pointed at a new cluster, and the hub was down for about an hour
    2i2c2025
  2. IL-0197 Moving a three-node Elasticsearch StatefulSet to new persistent volumes split the cluster in two, because the Helm chart hardcoded cluster.initial_master_nodes to elasticsearch-0
    Palark2021
  3. IL-0506 UC Berkeley DataHub logins failed and students got empty home directories after the JupyterHub database was overwritten
    UC Berkeley2017

Capacity exhaustion4 cases

Component
Persistent volumes and stateful sets4Cloud block storage1In-memory cache1Cloud controller manager1Database1Helm1JupyterHub1
Contributing factor
Monitoring blind spot2Unverified assumption1Change during peak hours1Unknown limit1Limited debug access1No alert on limit1No capacity planning1No runbook1Retry storm1Single point of failure1Unrepresentative staging1
Organisation
2i2c1BC Government1freedesktop.org1Magic Pages1
  1. IL-0067 Redis filled its persistent volume, corrupted its append-only file and crash-looped on every restart
    BC Government2025
  2. IL-0085 A control plane node of Magic Pages' Kubernetes cluster on Hetzner Cloud ran out of memory, the cluster's own calls then used up the Hetzner API rate limit, and all customer websites were down for about 7.5 hours
    Magic Pages2025
  3. IL-0500 UToronto JupyterHub logins and server starts returned 500 errors when the hub's sqlite database disk filled up
    2i2c2022
  4. IL-0518 gitlab.freedesktop.org on GKE ran short of volume space and a planned 30-minute storage upgrade became a 4-hour outage
    freedesktop.org2018

Dependency failure1 case

Component
DatabasePersistent volumes and stateful sets
Contributing factor
Harmful manual actionMonitoring blind spotSingle point of failure
Organisation
Magic Pages
  1. IL-0489 Rejoining a failed node of Magic Pages' three-node MySQL cluster broke its metadata and took customer websites down
    Magic Pages2025

Certificate or credential expiry1 case

Component
Cluster cloud identityCNI pluginNode poolPersistent volumes and stateful sets
Contributing factor
Monitoring blind spot
Organisation
Jenkins project
  1. IL-0521 AKS pods could not mount volumes and new nodes got no CNI network interface after the cluster service principal expired
    Jenkins project2021

Software defect2 cases

Component
Persistent volumes and stateful sets2Database1etcd1Ingress controller1Kubernetes API server1Container memory limit1
Contributing factor
Unverified assumption1Unreviewed defaults1No staged rollout1Single point of failure1
Organisation
Nordstrom1Palark1
  1. IL-0196 Adding a second Redis replica and lowering memory limits in one RedisFailover change restarted the only Redis pod that held data, and Sentinel promoted the empty new replica, erasing the data
    Palark2020
  2. IL-0072 High etcd write latency made the API servers return stale data, and the production cluster flipped between two views of its state
    Nordstrom2017

Cloud provider outage1 case

Component
DatabaseNode root diskPersistent volumes and stateful sets
Contributing factor
Unverified assumptionUnpatched known bugOverlapping failures
Organisation
Grafana Labs
  1. IL-0193 A GCP persistent disk degradation took down a Cassandra StatefulSet whose OrderedReady policy blocked recovery, and Metrictank could not reconnect to the recreated pods, causing a 23-hour Graphite query outage
    Grafana Labs2019

unknown1 case

Component
Node poolPersistent volumes and stateful sets
Contributing factor
Pages not delivered
Organisation
Binder Project
  1. IL-0402 mybinder.org launches dropped to zero for hours because a JupyterHub pod on a lost GKE node kept its hub database PVC
    Binder Project2018