Incident Library
  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases32 cases

17 to 21 of 32

  1. IL-0060 Kubernetes 1.24 upgrade removed the master node label that Calico route reflectors selected on, dropping all pod routes and taking Reddit down for 314 minutes
    Reddit2023
    Symptoms

    About two minutes after an engineer started the Kubernetes 1.23 to 1.24 upgrade of the cluster running Reddit's legacy stack, just after 19:00 UTC, the site stopped. The CDN edge request graph, which is kept separate from the cluster, showed traffic by status collapsing. Users saw the error page, a homepage with no posts, or plain errors.

    Service outageHigh error rateDNS resolution failuresMonitoring outagePods stuck in ContainerCreatingControl plane lag
    Trigger

    An in-place upgrade of the legacy cluster from Kubernetes 1.23 to 1.24; in 1.24 the node-role.kubernetes.io/master label is removed even from running clusters.

    Root cause
    Breaking change in an upgrade
    Component
    CalicoKubernetes API serverAdmission webhooketcd
    Contributing factor
    Configuration not in codeUnrepresentative stagingUntested backupMonitoring blind spotSingle point of failure
    Detection

    The site broke about two minutes into the upgrade; an incident was opened immediately and responders were on the call within three minutes, working from CDN edge graphs because the cluster's own metrics were gone.

    Remediation
    • deleted the Open Policy Agent webhook configurations to stop API server timeouts
    • terminated all worker nodes of the cluster
    • restored the control plane from the pre-upgrade backup onto the first control plane node
    • switched off the cluster autoscaler until the control plane was stable
    • generated new TLS certificates so replacement control plane nodes could join etcd
    • brought traffic back in steps from 1% to 100%
    • restarted legacy services that had been stopped by hand
    Action items
    • Codify cluster configuration so every choice has a record, and document in detail what cannot be codified
    • Standardise components across clusters and replace bespoke configurations
    • Build tooling to manage the lifecycle of whole clusters so they can be recreated or replicated
    • Add pre-production clusters that can be tested with confidence
    • Improve the backup restore process and make cluster upgrades safe
    Lessons

    Configuration that exists only in a live cluster, edited by hand through a special tool, can be forgotten, as the route reflector setup was, and then break on an upgrade nobody connects it to. A Kubernetes upgrade cannot be reverted without a restore, so the restore procedure has to be kept current and practised. Consistent, codified clusters are what make upgrades testable; bespoke clusters keep producing outages.

    View case
  2. IL-0113 The managed GKE control plane of Webhook Relay's main cluster failed because its API server certificates had never been rotated, and a manual credential rotation reported success without taking effect
    Webhook Relay2022
    Symptoms

    At 12:00 EST the health check system reported that the backend APIs, the frontend and the tunnels in the EU region were down. The team first took it for a transient error, such as an interrupted client request or a server evicted for memory or CPU pressure, but the main dashboard was offline as well.

    Service outageControl plane lagControl plane unresponsiveStale or inconsistent API readsCredentials unavailable
    Root cause
    Certificate or credential expiry
    Component
    GKE control planeKubernetes API serverNode poolRBAC and service accounts
    Contributing factor
    Limited debug accessSingle point of failure
    Detection

    The health check system sent a notification that the service was down; the team then found the main dashboard offline.

    Remediation
    • recreated the node pools by hand after the credential rotation failed to recreate them
    • duplicated the main cluster's services into a newly created GKE cluster
    • deleted the Kubernetes Service records in the old cluster so its load balancers could be attached to the new cluster
    • detached the StatefulSet from the old cluster and switched all writes to the main disk
    • exported the data from the temporary ClickHouse instance and inserted it into the main instance
    Action items
    • Change the deployment manifests so services can be switched between clusters quickly without a complicated persistent disk migration
    Lessons

    Recovery of existing infrastructure needs a time limit. Throughout the outage the old cluster looked about to recover, and the reluctance to give up on the salvage effort kept the team from starting over with a new cluster much sooner, which would have been the right call. On older GKE clusters, check the validity of the cluster CA certificate, for example by decoding masterAuth.clusterCaCertificate from gcloud container clusters describe and reading it with openssl x509. GKE's multiple availability zones made no difference in this failure.

    View case
  3. IL-0184 JupyterHub's KubeSpawner could not create containers and kubectl was refused because the API server and other internal certificates of a kubeadm cluster had expired
    dbhang2021
    Symptoms

    KubeSpawner, which starts JupyterHub user containers on Kubernetes, failed to create containers with an error the author had never seen, saying an SSL certificate had expired.

    Control plane unresponsiveDeploys failing
    Root cause
    Certificate or credential expiry
    Component
    Kubernetes API serverkubelet
    Contributing factor
    Unknown limitNo alert on limit
    Detection

    KubeSpawner failed to create a container and logged an SSL certificate verification error.

    Remediation
    • checked certificate expiry with kubeadm alpha certs check-expiration
    • moved the old certificates and /etc/kubernetes/*.conf files to a backup directory
    • regenerated all certificates with kubeadm init phase certs all
    • regenerated the kubeconfig files with kubeadm init phase kubeconfig all
    • copied /etc/kubernetes/admin.conf to ~/.kube/config and restarted kubelet
    • restarted the kube-system containers with docker restart so they picked up the new credentials
    Lessons

    kubeadm clusters run on certificates that expire, and when the API server certificate expires kubectl cannot be used to fix it, so renewal has to happen on the control plane node with kubeadm. After renewal, the kubeconfig files and the running control plane containers must also pick up the new certificates, or kubectl returns Unauthorized.

    View case
  4. IL-0108 An Alcide admission controller applied to the production cluster during a workshop made create and update requests on Deployments time out
    DFDS2020
    Symptoms

    At 14:41 CET on 4 November 2020 users reported timeouts when deploying to Hellman, the production Kubernetes cluster. The timeouts were confirmed at 15:04.

    Deploys failingControl plane lag
    Trigger

    During an Alcide workshop that began at 13:00 CET, an Alcide admission controller was applied to the production cluster at 13:13 in the belief that it would not disrupt anything.

    Root cause
    Configuration change
    Component
    Admission webhookKubernetes API server
    Contributing factor
    No configuration testsUnverified assumptionMonitoring blind spot
    Detection

    Users reported timeouts when deploying to the Hellman cluster at 14:41 CET; the timeouts were verified at 15:04.

    Remediation
    • toggled the Alcide admission controller to verify that it caused the timeouts
    • permanently deleted the Alcide admission controller
    Action items
    • Monitor Kubernetes deployments for timeouts
    • Monitor 5xx status codes from the Kubernetes API server
    Lessons

    New components are applied in sandbox clusters, not in production. The team needs enough Alcide knowledge to maintain the component until it is fully deprecated.

    View case
  5. IL-0458 Toolforge Kubernetes API server failed to start after a restart, logging "etcd cluster is unavailable or misconfigured"
    Wikimedia Foundation2019
    Symptoms

    On 10 September 2019 the Kubernetes API server of Wikimedia's Toolforge cluster did not come back after the maintain-kubeusers process restarted it at 18:30. To monitoring and to users the whole cluster looked down: Toolforge users could not launch tools with the webservice command on the Kubernetes backend or use kubectl, and an Icinga check against the "All k8s nodes healthy" endpoint started alerting at 18:45.

    Control plane unresponsiveService outageTLS certificate errors
    Trigger

    At 18:30 the maintain-kubeusers process restarted the Kubernetes API server, which forced it to reconnect to etcd with a CA certificate that Puppet changes the night before had left mismatched with the etcd servers.

    Root cause
    Configuration change
    Component
    Kubernetes API serveretcdConfiguration management tool
    Contributing factor
    Change with a delayed effectHarmful manual actionNo runbook
    Detection

    Automated monitoring: an Icinga alert on the toolschecker check http://checker.tools.wmflabs.org/k8s/nodes/ready, about "All k8s nodes healthy", fired at 18:45.

    Remediation
    • Rebooted the Kubernetes master and the etcd cluster, which did not help
    • Took a backup of etcd and reduced the etcd cluster to one node to rule out raft election flaps
    • Brought etcd back to full capacity through a full disaster recovery from the backup
    • Swapped the API server's CA certificate for the one from the etcd servers, which restored services until Puppet reverted it
    • Placed the etcd Puppet CA certificate in /var/lib/puppet/client/ssl/certs/ca.pem so Puppet kept the right CA
    Action items
    • Document the etcd cluster processes
    • Upgrade Toolforge Kubernetes to improve etcd and Kubernetes error reporting and stop needing an API server restart to add a tool
    • Make kube2proxy more resilient to API server failures
    • Audit the Puppet CA certificates so the tools project has no more than one
    • Make the Kubernetes control plane highly available
    Lessons

    Components that often misbehave, such as etcd, drew the responders to the wrong fix, and working on etcd instead of the CA certificates made the outage worse. Puppet-managed certificates are risky, and very old Kubernetes versions log errors that are hard to read. The failure was hard to find because it surfaced at an API server restart rather than at the Puppet change, because the etcd cluster had long-standing undocumented quirks, and because no disaster recovery or node replacement procedure was written down for etcd.

    View case
Check your cluster for this

These 32 cases involve Kubernetes API server and etcd. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause9

Credential exposure1 case

Component
Kubernetes API serverRBAC and service accounts
Contributing factor
Excessive blast radius
Organisation
Unnamed cryptocurrency exchange
  1. IL-0069 Attackers stole a high-privilege service account token from a production Kubernetes cluster and used it to reach a cryptocurrency exchange's financial systems
    Unnamed cryptocurrency exchange2026

Capacity exhaustion10 cases

Component
Kubernetes API server10etcd5Telemetry agent2CI system1Cilium1CoreDNS1Credential broker for pods1Ingress controller1kubelet1Managed Kubernetes control plane and provider API1Container memory limit1Cluster authentication provider1
Contributing factor
Unreviewed defaults5Excessive blast radius2Unknown limit2Harmful manual action2Retry storm2Single point of failure2Unverified assumption1Automation amplification1Ignored early signal1Limited debug access1Monitoring blind spot1No capacity planning1No staged rollout1
Organisation
Palark2Zalando2Civis Analytics1Fullstaq1IONOS Cloud1Linode1Render1Scaleway1
  1. IL-0464 IONOS Managed Kubernetes control planes went unreachable for minutes at a time as their shared etcd grew and fragmented
    IONOS Cloud2026
  2. IL-0204 An etcd memory spike overloaded the control plane of a Render Frankfurt cluster, and DNS then failed because CoreDNS ran on the rebooted control plane nodes and could not start without the API server
    Render2024
  3. IL-0168 Simultaneous cluster auto-upgrades overloaded Scaleway's Kubernetes API, and a node authentication retry loop kept Kapsule and Kosmos nodes NotReady in FR-PAR
    Scaleway2024
  4. IL-0195 The kube-apiservers of a cluster with more than 200 nodes ran out of memory and crashed one after another under mass LIST requests from Cilium agents
    Palark2023
  5. IL-0165 Some LKE users in Frankfurt could not reach their kube-apiservers with kubectl after backend services failed under memory constraints
    Linode2023
  6. IL-0194 After a kube-apiserver restart, pod LIST requests from every pod of a Vector DaemonSet nearly tripled etcd memory and restarted all master nodes
    Palark2023
  7. IL-0002 CI builder pods could not get AWS IAM credentials because kubelet fell minutes behind its API rate limit
    Zalando2019
  8. IL-0076 Kubernetes masters went down after Dex filled etcd with AuthRequest objects and deleting its namespace started an endless list-and-timeout loop
    Fullstaq2019
  9. IL-0075 Kubernetes API servers pegged on CPU when the cluster grew to about 290 nodes, traced to Datadog agents polling the API
    Civis Analytics2019
  10. IL-0031 Kubernetes API server OOM-killed, leaving the Skipper ingress without routes; the talk points to a misconfigured CronJob manifest
    Zalando2018

Certificate or credential expiry3 cases

Component
Kubernetes API server3kubelet2Calico1etcd1GKE control plane1Node pool1RBAC and service accounts1
Contributing factor
Unreviewed defaults1Unknown limit1Limited debug access1No alert on limit1Single point of failure1
Organisation
ckck8031dbhang1Webhook Relay1
  1. IL-0183 kubeadm's one-year leaf certificates expired, so kube-apiserver could not reach etcd, restarted in a loop and left the cluster unreachable
    ckck8032026
  2. IL-0113 The managed GKE control plane of Webhook Relay's main cluster failed because its API server certificates had never been rotated, and a manual credential rotation reported success without taking effect
    Webhook Relay2022
  3. IL-0184 JupyterHub's KubeSpawner could not create containers and kubectl was refused because the API server and other internal certificates of a kubeadm cluster had expired
    dbhang2021

Software defect5 cases

Component
Kubernetes API server5etcd3Cloud load balancer1GPU driver1Ingress controller1kubelet1Linkerd1Linux kernel1Persistent volumes and stateful sets1Policy engine1
Contributing factor
Monitoring blind spot2Single point of failure2Unreviewed defaults1Change with a delayed effect1Ignored early signal1Unpatched known bug1Limited debug access1Harmful manual action1No alert on limit1Overlapping failures1Ineffective rollback1
Organisation
Barush Mendez1Marcin Cuber1Monzo1Nordstrom1Zalando1
  1. IL-0138 A failed Nvidia driver build left broken kernel packages on the only Kubernetes master node, which hit a kernel panic on reboot after a power outage and took the homelab cluster down for six days
    Barush Mendez2025
  2. IL-0077 EKS cluster went read-only for over seven hours after Kyverno ephemeralreports filled the etcd database past its size limit
    Marcin Cuber2024
  3. IL-0030 Most nodes of a cluster went NotReady together for 15 minutes while the API server sat behind an AWS ELB
    Zalando2019
  4. IL-0072 High etcd write latency made the API servers return stale data, and the production cluster flipped between two views of its state
    Nordstrom2017
  5. IL-0003 linkerd lost Kubernetes service discovery after an etcd reconfiguration, and restarting every linkerd instance turned failed payments into a full platform outage
    Monzo2017

Configuration change7 cases

Component
Kubernetes API server7Admission webhook2etcd2Configuration management tool1Client-side DNS resolver and cache1GKE control plane1kubelet1RBAC and service accounts1Container registry and its credentials1Telemetry agent1
Contributing factor
Monitoring blind spot4Excessive blast radius3Automation amplification2Limited debug access2Unverified assumption1Change with a delayed effect1Cache-masked failure1Harmful manual action1No configuration tests1No health probes1No runbook1No safeguard on destructive change1No staged rollout1Unrepresentative staging1
Organisation
OpenAI2Aqua Security1DFDS1DigitalOcean1Jetstack1Wikimedia Foundation1
  1. IL-0005 A new telemetry service overloaded the Kubernetes API servers of large clusters and broke DNS-based service discovery for all OpenAI services
    OpenAI2024
  2. IL-0118 A global change to Kubernetes namespace labels made the networking layer recompute metadata and overwhelmed the control planes of three of OpenAI's largest GPU clusters
    OpenAI2024
  3. IL-0152 An incorrect configuration applied during maintenance made an internal infrastructure cluster automatically delete the control planes of 200 DOKS clusters in FRA1
    DigitalOcean2024
  4. IL-0068 Anonymous access to the Kubernetes API server used to plant a ClusterRoleBinding backdoor and a cryptominer DaemonSet
    Aqua Security2023
  5. IL-0108 An Alcide admission controller applied to the production cluster during a workshop made create and update requests on Deployments time out
    DFDS2020
  6. IL-0458 Toolforge Kubernetes API server failed to start after a restart, logging "etcd cluster is unavailable or misconfigured"
    Wikimedia Foundation2019
  7. IL-0008 An unresponsive OPA validating webhook crash-looped the API server during a GKE master upgrade, and node auto-repair then recreated every node
    Jetstack2019

Cloud provider outage3 cases

Component
Kubernetes API server3AKS control plane1GKE control plane1kubelet1
Contributing factor
Limited debug access2No capacity planning1No headroom1Overlapping failures1
Organisation
Binder Project1Namespace1UC Berkeley1
  1. IL-0084 Packet drops on a bare metal provider's vSwitch forced Namespace to rebuild its fra3 region and left global capacity short, and the next day slow NVMe writes on a control plane node drove the Kubernetes API server to very high CPU
    Namespace2024
  2. IL-0469 The managed Azure Kubernetes API server for the data100 hub stopped answering, so the stopped hub pod was not recreated
    UC Berkeley2019
  3. IL-0505 kubectl logs failed on GKE with "No SSH tunnels currently open" and mybinder.org build logs stopped reaching users
    Binder Project2019

DNS or network fault1 case

Component
Kubernetes API serverkubelet
Contributing factor
–
Organisation
DigitalOcean
  1. IL-0154 Line cards on an SFO3 core router rebooted together and halved backbone capacity, and DOKS nodes whose guest networking did not recover stayed NotReady with degraded API servers until all affected nodes in the SFO3 DOKS fleet were recycled
    DigitalOcean2024

Breaking change in an upgrade1 case

Component
Admission webhookCalicoetcdKubernetes API server
Contributing factor
Untested backupConfiguration not in codeMonitoring blind spotSingle point of failureUnrepresentative staging
Organisation
Reddit
  1. IL-0060 Kubernetes 1.24 upgrade removed the master node label that Calico route reflectors selected on, dropping all pod routes and taking Reddit down for 314 minutes
    Reddit2023

unknown1 case

Component
etcdKubernetes API serverkubelet
Contributing factor
Unreviewed defaultsIgnored early signalHarmful manual actionMonitoring blind spotSingle point of failure
Organisation
NRE Labs
  1. IL-0059 Single-master Kubernetes control plane kept dying because the kubelet restarted a healthy etcd on a failing kubeadm liveness probe
    NRE Labs2018