Incident Library
Applied filters FactorNo safeguard on destructive changeDomainKubernetesClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases13 cases

13 to 13 of 13

  1. IL-0506 UC Berkeley DataHub logins failed and students got empty home directories after the JupyterHub database was overwritten
    UC Berkeley2017
    Symptoms

    At about 1 AM on 9 February 2017, UC Berkeley's DataHub was reported down: users trying to log in got a proxy error. The hub pod was running, but its log was full of sqlite errors.

    Service outageData lossMisrouted requests
    Trigger

    While debugging, an operator ran `.save jupyterhub.sqlite` in sqlite3 inside the hub pod without having attached the database, which replaced the JupyterHub database file with an empty one.

    Root cause
    Operator error
    Component
    DatabasePersistent volumes and stateful setsJupyterHub
    Contributing factor
    No safeguard on destructive changeUnsafe failure behaviourHarmful manual action
    Detection

    DataHub was reported down at about 1 AM, when users got a proxy error at login. At 1:12 AM a user reported that he could log in but all his notebooks were gone.

    Remediation
    • Scaled the hub to 0 replicas
    • Rebuilt the JupyterHub sqlite database by matching Kubernetes PVC names back to user ids
    • Set cookie_id to null instead of a dummy value and attached the rebuilt database to the hub
    • Changed the cookie secret in the hub pod and restarted it to log all users out
    • Deleted all user pods to log everyone out
    • Ran a script that filled in cookie_id for all users, restarted the hub and deleted all user pods again
    • Dropped the new users with a '.' in their name and renamed the older users from '-' to '.'
    Action items
    • Report upstream that JupyterHub uses only part of the cookie secret and discards the rest
    • Report upstream that the cookie_id column of the users table should have UNIQUE and NOT NULL constraints
    • Ask KubeSpawner to support username hashes in PVC and pod names so the names stay the same when the database is deleted
    • Ask OAuthenticator to support using the id from the Google authenticator as the user id, giving stable user ids
    • Switch to Google Cloud SQL, a hosted and managed MySQL database
    • Take regular and tested backups of the database
    • Start an operational FAQ of things to do and not to do
    • Set up better monitoring and paging
    • Document escalation procedures explicitly
    View case
Check your cluster for this

These 13 cases involve Cloud load balancer and Ingress controller. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause3

Software defect2 cases

Component
Argo CD1etcd1Node pool1
Contributing factor
No safeguard on destructive change2Unverified assumption1Unreviewed defaults1Harmful manual action1
Organisation
Argo CD users1Grafana Labs1
  1. IL-0058 Argo CD deleted namespaces shared by several applications, leaving about 97 applications stuck Missing and OutOfSync
    Argo CD users2026
  2. IL-0190 Cortex distributors kept a stuck connection to a dead etcd leader after a Kubernetes upgrade terminated its node, failing writes to Hosted Prometheus for 12 minutes
    Grafana Labs2020

Operator error6 cases

Component
Ingress controller3Database2Persistent volumes and stateful sets2Terraform2CI system1Cloud load balancer1Credential broker for pods1external-dns1Jobs and CronJobs1JupyterHub1Model hub1Prometheus1VPC route tables and firewall rules1
Contributing factor
No safeguard on destructive change6Harmful manual action1No runbook1No headroom1Overlapping failures1Deferred upgrades1Unsafe failure behaviour1
Organisation
UK Ministry of Justice22i2c1GitLab1OpenAI1UC Berkeley1
  1. IL-0095 An engineer deleted the prod namespace of the LEAP JupyterHub while believing the kubeconfig pointed at a new cluster, and the hub was down for about an hour
    2i2c2025
  2. IL-0244 A queued pull request destroyed the EKS subnet route table associations and Cloud Platform services went offline
    UK Ministry of Justice2024
  3. IL-0207 A routine task run from an engineer's local environment switched to production and deleted GitLab.com's Consul release, so Patroni lost Consul and GitLab.com returned 5xx errors to all traffic
    GitLab2022
  4. IL-0115 One of OpenAI's three production AKS clusters went offline through a human error, and rebuilding it took 10½ hours because an AKS Kubernetes upgrade broke the ingress and AKS cannot be downgraded
    OpenAI2022
  5. IL-0089 A test cluster deletion script run with the kube context pointing at the live-1 production cluster deleted its ingress controller, monitoring, logging, cert-manager, kiam and external-dns
    UK Ministry of Justice2020
  6. IL-0506 UC Berkeley DataHub logins failed and students got empty home directories after the JupyterHub database was overwritten
    UC Berkeley2017

Configuration change5 cases

Component
Cloud load balancer2Configuration templating2Argo CD1CI system1CNI plugin1etcd1GKE control plane1Kubernetes API server1Node pool1Persistent volumes and stateful sets1Terraform1
Contributing factor
No safeguard on destructive change5Unverified assumption3Automation amplification2Excessive blast radius2No staged rollout2Missed in review2Monitoring blind spot1No runbook1
Organisation
DigitalOcean1Grafana Labs1InfluxData1Jenkins project1Spotify1
  1. IL-0192 A Crossplane composition merge-policy change for a TLS policy update switched Kubernetes Services from LoadBalancer to ClusterIP and deleted the load balancers behind about 25% of Grafana Cloud services
    Grafana Labs2025
  2. IL-0152 An incorrect configuration applied during maintenance made an internal infrastructure cluster automatically delete the control planes of 200 DOKS clusters in FRA1
    DigitalOcean2024
  3. IL-0473 Jenkins infrastructure went fully down after a pod IP range fix deleted every AKS node pool and could not recreate them
    Jenkins project2023
  4. IL-0009 A duplicated generated name made Argo CD replace the InfluxDB Cloud production Application and delete the whole production environment
    InfluxData2022
  5. IL-0061 A Terraform pull request merged out of order deleted a production GKE cluster, and recreating it deleted a second one
    Spotify2019