Incident Library
  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases24 cases

24 to 24 of 24

  1. IL-0506 UC Berkeley DataHub logins failed and students got empty home directories after the JupyterHub database was overwritten
    UC Berkeley2017
    Symptoms

    At about 1 AM on 9 February 2017, UC Berkeley's DataHub was reported down: users trying to log in got a proxy error. The hub pod was running, but its log was full of sqlite errors.

    Service outageData lossMisrouted requests
    Trigger

    While debugging, an operator ran `.save jupyterhub.sqlite` in sqlite3 inside the hub pod without having attached the database, which replaced the JupyterHub database file with an empty one.

    Root cause
    Operator error
    Component
    DatabasePersistent volumes and stateful setsJupyterHub
    Contributing factor
    No safeguard on destructive changeUnsafe failure behaviourHarmful manual action
    Detection

    DataHub was reported down at about 1 AM, when users got a proxy error at login. At 1:12 AM a user reported that he could log in but all his notebooks were gone.

    Remediation
    • Scaled the hub to 0 replicas
    • Rebuilt the JupyterHub sqlite database by matching Kubernetes PVC names back to user ids
    • Set cookie_id to null instead of a dummy value and attached the rebuilt database to the hub
    • Changed the cookie secret in the hub pod and restarted it to log all users out
    • Deleted all user pods to log everyone out
    • Ran a script that filled in cookie_id for all users, restarted the hub and deleted all user pods again
    • Dropped the new users with a '.' in their name and renamed the older users from '-' to '.'
    Action items
    • Report upstream that JupyterHub uses only part of the cookie secret and discards the rest
    • Report upstream that the cookie_id column of the users table should have UNIQUE and NOT NULL constraints
    • Ask KubeSpawner to support username hashes in PVC and pod names so the names stay the same when the database is deleted
    • Ask OAuthenticator to support using the id from the Google authenticator as the user id, giving stable user ids
    • Switch to Google Cloud SQL, a hosted and managed MySQL database
    • Take regular and tested backups of the database
    • Start an operational FAQ of things to do and not to do
    • Set up better monitoring and paging
    • Document escalation procedures explicitly
    View case
Check your cluster for this

These 24 cases involve etcd and Kubernetes API server. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause11

Capacity exhaustion10 cases

Component
CoreDNS2Database2etcd2Ingress controller2Kubernetes API server2Container memory limit2Admission webhook1CI system1Cluster autoscaler1Horizontal pod autoscaler1JupyterHub1Karpenter1NAT gateway1Cluster authentication provider1Pod scheduling and placement1Prometheus1Container registry and its credentials1VPC subnet or pod address range1Telemetry agent1AWS VPC CNI1
Contributing factor
Harmful manual action10Excessive blast radius3Unreviewed defaults3Unverified assumption2Automation amplification2No alert on limit2No resource requests or limits2Retry storm2Ineffective rollback2Single point of failure2Ignored early signal1Unknown limit1Limited debug access1Monitoring blind spot1No capacity planning1No headroom1Partial rollout1
Organisation
Binder Project1Buildkite1Datadog1DFDS1Fullstaq1GitHub1Neon1Render1UK Ministry of Justice1이언철1
  1. IL-0471 GitHub Actions runs failed or sat queued for 9 hours after a routine deployment's pod replacement saturated capacity
    GitHub2026
  2. IL-0470 Buildkite notifications were delayed after Prometheus storage filled and EKS notification workers could not autoscale
    Buildkite2026
  3. IL-0006 VPC subnets ran out of IP addresses because the AWS VPC CNI kept IPs on full nodes, and WARM_IP_TARGET changes then blocked pod starts after every pod deletion
    Neon2025
  4. IL-0204 An etcd memory spike overloaded the control plane of a Render Frankfurt cluster, and DNS then failed because CoreDNS ran on the rebooted control plane nodes and could not start without the API server
    Render2024
  5. IL-0182 Too few CoreDNS replicas on EKS were overloaded during traffic spikes, raising latency and cutting the network for task pods, and the over-provisioned, tightly spread CoreDNS pods added later blocked Karpenter consolidation
    이언철2024
  6. IL-0110 A Grafana Agent pod's high memory use and crash-looping Hangfire pods drove a Kubernetes node into system OOM and NotReady, twice in one day
    DFDS2024
  7. IL-0093 Ingress controller pods crash-looped one after another because the leader pod had exhausted its shared memory
    UK Ministry of Justice2020
  8. IL-0076 Kubernetes masters went down after Dex filled etcd with AuthRequest objects and deleting its namespace started an endless list-and-timeout loop
    Fullstaq2019
  9. IL-0037 Image registry rate-limited a whole region after a crash-looping DaemonSet with imagePullPolicy Always pulled through three NAT instances
    Datadog2019
  10. IL-0503 mybinder.org launches failed after a GKE node ran out of memory and its nginx-ingress proxy pod crash-looped
    Binder Project2017

Software defect2 cases

Component
Argo CD1etcd1Kubernetes API server1kubelet1Linkerd1
Contributing factor
Harmful manual action2Unverified assumption1Monitoring blind spot1No safeguard on destructive change1Overlapping failures1Ineffective rollback1Single point of failure1
Organisation
Argo CD users1Monzo1
  1. IL-0058 Argo CD deleted namespaces shared by several applications, leaving about 97 applications stuck Missing and OutOfSync
    Argo CD users2026
  2. IL-0003 linkerd lost Kubernetes service discovery after an etcd reconfiguration, and restarting every linkerd instance turned failed payments into a full platform outage
    Monzo2017

Cloud provider outage1 case

Component
Cloud load balancerCluster autoscalerGKE control planeHorizontal pod autoscaler
Contributing factor
Harmful manual actionNo headroomOverlapping failuresRetry storm
Organisation
Twingate
  1. IL-0542 Twingate authentication and authorization requests failed at higher rates after traffic moved onto a smaller GKE cluster
    Twingate2025

Dependency failure1 case

Component
DatabasePersistent volumes and stateful sets
Contributing factor
Harmful manual actionMonitoring blind spotSingle point of failure
Organisation
Magic Pages
  1. IL-0489 Rejoining a failed node of Magic Pages' three-node MySQL cluster broke its metadata and took customer websites down
    Magic Pages2025

unknown2 cases

Component
Calico1Container runtime1etcd1Kubernetes API server1kubelet1Multus meta-CNI1
Contributing factor
Harmful manual action2Unverified assumption1Unreviewed defaults1Ignored early signal1Monitoring blind spot1Single point of failure1
Organisation
Fyra Labs1NRE Labs1
  1. IL-0087 After a cold reboot of RKE2 nodes, pods could not get IP addresses because Multus could not reach the Kubernetes API, and further reboots spread the failure to every node
    Fyra Labs2023
  2. IL-0059 Single-master Kubernetes control plane kept dying because the kubelet restarted a healthy etcd on a failing kubeadm liveness probe
    NRE Labs2018

Breaking change in an upgrade2 cases

Component
Admission webhook1CDN and edge network1CoreDNS1Client-side DNS resolver and cache1Helm1Ingress controller1kube-proxy1kubelet1
Contributing factor
Harmful manual action2Monitoring blind spot2Unverified assumption1Cache-masked failure1Partial rollout1Ineffective rollback1
Organisation
CircleCI1Jenkins project1
  1. IL-0230 kube-proxy sync failures during a Kubernetes upgrade corrupted node iptables, and CircleCI jobs stopped starting
    CircleCI2023
  2. IL-0522 An ingress-nginx chart upgrade deleted the jenkins.io Ingress and Fastly served HTTP/503 errors for hours
    Jenkins project2021

Configuration change2 cases

Component
Configuration management tool1Container runtime1etcd1Kubernetes API server1Node launch template1Node pool1Telemetry agent1Terraform1
Contributing factor
Harmful manual action2No runbook2Automation amplification1Unreviewed defaults1Change with a delayed effect1No configuration tests1Unrepresentative staging1
Organisation
DFDS1Wikimedia Foundation1
  1. IL-0109 A retried node rollover on a DFDS EKS cluster set the Auto Scaling group back to 15 instances, and the default termination policy removed every healthy worker node, leaving the cluster with no nodes
    DFDS2023
  2. IL-0458 Toolforge Kubernetes API server failed to start after a restart, logging "etcd cluster is unavailable or misconfigured"
    Wikimedia Foundation2019

Faulty deployment1 case

Component
Container CPU limitNode root disk
Contributing factor
Unknown limitLimited debug accessHarmful manual actionNo staged rolloutOverlapping failuresIneffective rollbackUnrepresentative stagingGap in automated tests
Organisation
Gitpod
  1. IL-0591 Gitpod workspaces in all regions became IO bound and unresponsive for days after the ws39 workspace cluster release
    Gitpod2022

Certificate or credential expiry1 case

Component
cert-managerSecrets store
Contributing factor
Unverified assumptionHarmful manual actionMonitoring blind spotSingle point of failure
Organisation
GoCardless
  1. IL-0460 Vault served an expired TLS certificate and almost every GoCardless service pod crash-looped, a 22-minute full outage
    GoCardless2020

DNS or network fault1 case

Component
Container memory limitContainer registry and its credentials
Contributing factor
Harmful manual actionOverlapping failuresSingle point of failure
Organisation
GitHub
  1. IL-0199 A non-essential container exceeding its memory limit made GitHub's production pods unavailable, and their restarts failed because a DNS maintenance change had left the clusters unable to reach the image registry
    GitHub2020

Operator error1 case

Component
DatabaseJupyterHubPersistent volumes and stateful sets
Contributing factor
Harmful manual actionNo safeguard on destructive changeUnsafe failure behaviour
Organisation
UC Berkeley
  1. IL-0506 UC Berkeley DataHub logins failed and students got empty home directories after the JupyterHub database was overwritten
    UC Berkeley2017