Incident Library
  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases52 cases

21 to 25 of 52

  1. IL-0068 Anonymous access to the Kubernetes API server used to plant a ClusterRoleBinding backdoor and a cryptominer DaemonSet
    Aqua Security2023
    Symptoms

    Aqua's research team Nautilus recorded the attack on one of its Kubernetes honeypots. What appeared in the cluster was a DaemonSet running the image kuberntesio/kube-controller:1.0.1 from Docker Hub on every node, using the nodes' CPU to mine Monero.

    Unauthorised access or a foreign workloadResources deleted
    Trigger

    An API server misconfigured to accept unauthenticated requests from anonymous users with privileges.

    Root cause
    Configuration change
    Component
    Kubernetes API serverRBAC and service accountsContainer registry and its credentials
    Contributing factor
    Monitoring blind spot
    Detection

    Aqua recorded the attacker's API requests on its honeypot; planted AWS access keys raised a beacon when the attacker tried to use them.

    Lessons

    Fixing an exposed API server is not enough once an attacker has been in: a ClusterRoleBinding to an attacker-created ServiceAccount keeps working after anonymous access is closed. Attackers name these objects and images after real Kubernetes components so they blend into the API audit logs and pass for legitimate deployments. Aqua recommends scanning clusters for vulnerabilities, exposed secrets and misconfigurations with a tool such as Trivy.

    View case
  2. IL-0060 Kubernetes 1.24 upgrade removed the master node label that Calico route reflectors selected on, dropping all pod routes and taking Reddit down for 314 minutes
    Reddit2023
    Symptoms

    About two minutes after an engineer started the Kubernetes 1.23 to 1.24 upgrade of the cluster running Reddit's legacy stack, just after 19:00 UTC, the site stopped. The CDN edge request graph, which is kept separate from the cluster, showed traffic by status collapsing. Users saw the error page, a homepage with no posts, or plain errors.

    Service outageHigh error rateDNS resolution failuresMonitoring outagePods stuck in ContainerCreatingControl plane lag
    Trigger

    An in-place upgrade of the legacy cluster from Kubernetes 1.23 to 1.24; in 1.24 the node-role.kubernetes.io/master label is removed even from running clusters.

    Root cause
    Breaking change in an upgrade
    Component
    CalicoKubernetes API serverAdmission webhooketcd
    Contributing factor
    Configuration not in codeUnrepresentative stagingUntested backupMonitoring blind spotSingle point of failure
    Detection

    The site broke about two minutes into the upgrade; an incident was opened immediately and responders were on the call within three minutes, working from CDN edge graphs because the cluster's own metrics were gone.

    Remediation
    • deleted the Open Policy Agent webhook configurations to stop API server timeouts
    • terminated all worker nodes of the cluster
    • restored the control plane from the pre-upgrade backup onto the first control plane node
    • switched off the cluster autoscaler until the control plane was stable
    • generated new TLS certificates so replacement control plane nodes could join etcd
    • brought traffic back in steps from 1% to 100%
    • restarted legacy services that had been stopped by hand
    Action items
    • Codify cluster configuration so every choice has a record, and document in detail what cannot be codified
    • Standardise components across clusters and replace bespoke configurations
    • Build tooling to manage the lifecycle of whole clusters so they can be recreated or replicated
    • Add pre-production clusters that can be tested with confidence
    • Improve the backup restore process and make cluster upgrades safe
    Lessons

    Configuration that exists only in a live cluster, edited by hand through a special tool, can be forgotten, as the route reflector setup was, and then break on an upgrade nobody connects it to. A Kubernetes upgrade cannot be reverted without a restore, so the restore procedure has to be kept current and practised. Consistent, codified clusters are what make upgrades testable; bespoke clusters keep producing outages.

    View case
  3. IL-0230 kube-proxy sync failures during a Kubernetes upgrade corrupted node iptables, and CircleCI jobs stopped starting
    CircleCI2023
    Symptoms

    On 14 March 2023 at about 18:00 UTC, communication errors rose on one CircleCI service, and pipelines for all customers began to start late until jobs did not run at all. Within an hour the errors had spread to many services and to every function of the application, including the CircleCI UI and API.

    Service outageIntermittent connection failuresDNS resolution failures
    Trigger

    At about 17:40 UTC on 14 March the infrastructure team resumed the upgrade of the main production Kubernetes cluster with a changed approach for some components, after a first attempt on 13 March had been rolled back. The ongoing upgrade left kubelet and kube-proxy at mismatched versions.

    Root cause
    Breaking change in an upgrade
    Component
    kube-proxykubeletCoreDNSClient-side DNS resolver and cache
    Contributing factor
    Partial rolloutIneffective rollbackMonitoring blind spotHarmful manual action
    Detection

    At 18:00 UTC engineers saw communication errors rise on one service, then job processing delays climbing and job queues backing up across several services. The kube-proxy "Sync failed" errors were found only in the analysis after the incident.

    Remediation
    • Restarted every node in the cluster so each node built a fresh iptables ruleset
    • Restarted a separate pool of dependencies so jobs using contexts or the machine executor ran again
    • Added capacity to work through the backlog of jobs
    • Restarted the RabbitMQ nodes to fix a few corrupted queues
    Action items
    • Improve the Kubernetes upgrade process to limit the risk of future upgrades
    • Add metrics and alerts for kube-proxy sync errors
    • Evaluate ways to make the RabbitMQ infrastructure more resilient
    Lessons

    No single trigger caused the incident. Because networking underlies everything in the cluster, Kubernetes upgrades carry inherent risk, and CircleCI's risk mitigation did not hold up when several events interacted.

    View case
  4. IL-0070 Critical pods on core nodes evicted when Prometheus began scraping about 200 nodes after a dask-gateway scale-up
    2i2c2023
    Symptoms

    The hub pod of the prod JupyterHub on the leap cluster (leap.2i2c.org) restarted unexpectedly during a large node scale-up, and a user opened a support ticket. The ingress-nginx and proxy pods kept running.

    Pod evictionPods Pending
    Trigger

    A user's dask-gateway workload added about 100 dask-worker nodes within a minute, and Prometheus started scraping the metrics exporters on about 200 nodes.

    Root cause
    Capacity exhaustion
    Component
    PrometheusNode pool
    Contributing factor
    No headroomExcessive blast radiusMonitoring blind spot
    Detection

    A user support ticket (2i2c Freshdesk ticket 414) about the leap hub; the engineer then looked at Grafana and the cluster's nodes and events.

    Action items
    • Configure CPU and memory requests for the hub and proxy pods (#2127; at least 100m CPU was proposed)
    • Make the Grafana OOMKiller dashboard work (#2213)
    • Move LEAP cloud infrastructure to shared nodes (#2209)
    • Update resource requests and limits, mainly for ingress-nginx and Prometheus (#2324)
    • Decide whether Prometheus should scrape dask worker pods (#2366)
    Lessons

    Prometheus load grows with the number of nodes, so a burst of autoscaled nodes can push a nearly full core node over the edge and evict the pods that matter most. Core pods that request 10m of CPU get a tenth of the CPU share of busy neighbours that request 100m. Kubernetes events expire after an hour, so evidence of OOM kills needs a dashboard that keeps it.

    View case
  5. IL-0116 A CNI configuration change incompatible with a hardware variant that exists only in production cut network connectivity in an OpenAI GPU cluster and took text-davinci-003 fully down
    OpenAI2023
    Symptoms

    At 4:53 pm Pacific on 25 January 2023 one of OpenAI's GPU clusters had a major outage, and the text-davinci-003 model went fully down.

    Service outageThroughput degradation
    Trigger

    A configuration change to the CNI (Container Network Interface) plugins, which provide network connectivity to containers, was deployed to a GPU cluster that contained a hardware variant the staging environment did not have.

    Root cause
    Configuration change
    Component
    CNI plugin
    Contributing factor
    Unrepresentative stagingMonitoring blind spot
    Detection

    Engineers immediately identified the problem as network connectivity, but it took nearly an hour to identify CNI as the cause.

    Remediation
    • put a fix for the CNI configuration change in place, which restored 80% of traffic immediately
    • deployed models back onto GPUs over about three more hours until capacity was fully stable
    Action items
    • Make staging environments contain every type of hardware configuration so they better represent production
    • Monitor and alert on the health of the CNI system directly
    • Fix the network and storage bottlenecks that limit how quickly many replicas of a model can be deployed
    Lessons

    A change that had been deployed to other clusters over 24 hours and deemed safe still broke a cluster with different hardware, so staging needs every hardware configuration that production has.

    View case
Check your cluster for this

These 52 cases involve kubelet and Kubernetes API server. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause9

Capacity exhaustion16 cases

Component
kubelet3Container memory limit3Node root disk3Telemetry agent3Cloud block storage2Container runtime2CoreDNS2Container CPU limit2Helm2Horizontal pod autoscaler2Node pool2Persistent volumes and stateful sets2Prometheus2In-memory cache1CI system1Cloud controller manager1conntrack table1Database1Client-side DNS resolver and cache1etcd1Liveness and readiness probes1Model inference server1Ingress controller1Istio1Kubernetes API server1Language runtime settings1
Contributing factor
Monitoring blind spot16Unreviewed defaults5Unknown limit5Excessive blast radius3Retry storm3Single point of failure3Unverified assumption2Ignored early signal2No headroom2Automation amplification1Change during peak hours1Limited debug access1Harmful manual action1No alert on limit1No caching1No capacity planning1No resource requests or limits1No runbook1Health check tied to a dependency1Unrepresentative staging1
Organisation
2i2c1Backpackr1Buildkite1Civis Analytics1DFDS1freedesktop.org1Kevin1Kubernetes project1Magic Pages1Microsoft1MUSINSA1Onkar Raghunath Shelke1Saltside1Temporal1UK Ministry of Justice1Zalando1
  1. IL-0071 Worker node running out of inodes while disk space looked fine, because containerd kept a full node_modules tree in each of several image layer snapshots
    Onkar Raghunath Shelke2026
  2. IL-0102 CoreDNS pods on a production EKS cluster ran out of memory under a burst of new pods, endpoints and nodes, taking down the Buildkite web UI, APIs and job dispatch
    Buildkite2026
  3. IL-0181 A traffic spike from an external community post pushed product API pods into CPU throttling under a 1-core limit, and an e-commerce product page slowed down while every pod stayed Running
    Kevin2026
  4. IL-0581 29CM product display servers lost pods and saturated Netty event loops on the first day of the Igu Week sale
    MUSINSA2025
  5. IL-0413 LLM inference engines timed out or ran out of resources under load in about 60% of 156 severe incidents at Microsoft
    Microsoft2025
  6. IL-0170 During a Lunar New Year event, traffic more than three times normal filled the nf_conntrack table on the EKS node running the Istio ingress gateway, dropping packets and taking down Backpackr's web and app services
    Backpackr2025
  7. IL-0085 A control plane node of Magic Pages' Kubernetes cluster on Hetzner Cloud ran out of memory, the cluster's own calls then used up the Hetzner API rate limit, and all customer websites were down for about 7.5 hours
    Magic Pages2025
  8. IL-0110 A Grafana Agent pod's high memory use and crash-looping Hangfire pods drove a Kubernetes node into system OOM and NotReady, twice in one day
    DFDS2024
  9. IL-0043 Application logs missing in Kibana after a newer Fluent Bit version changed its chunk drop strategy
    UK Ministry of Justice2023
  10. IL-0028 Temporal History service latency rose because its pods were CPU-throttled while CPU usage read below 60 percent
    Temporal2023
  11. IL-0070 Critical pods on core nodes evicted when Prometheus began scraping about 200 nodes after a dask-gateway scale-up
    2i2c2023
  12. IL-0401 Prow jobs on the k8s-prow-build GKE cluster failed and merges stalled as nodes hit IOPS throttling and kernel panics
    Kubernetes project2019
  13. IL-0075 Kubernetes API servers pegged on CPU when the cluster grew to about 290 nodes, traced to Datadog agents polling the API
    Civis Analytics2019
  14. IL-0001 Cluster-wide DNS outage after CoreDNS pods were OOM-killed again and again under a burst of DNS queries
    Zalando2019
  15. IL-0518 gitlab.freedesktop.org on GKE ran short of volume space and a planned 30-minute storage upgrade became a 4-hour outage
    freedesktop.org2018
  16. IL-0040 Parallel Helm installs saturated small gp2 root volumes on a kops cluster and left all three workers NotReady with the kubelet or Docker down
    Saltside2017

Dependency failure3 cases

Component
Cloud block storage1Database1Data center power and cooling1etcd1InfiniBand fabric1Node operating system image1Node pool1Persistent volumes and stateful sets1
Contributing factor
Monitoring blind spot3Single point of failure2Harmful manual action1No alert on limit1No runbook1Overlapping failures1
Organisation
FREE NOW1Magic Pages1Nebius1
  1. IL-0133 A storm shut down cooling at Nebius's us-central1 data center, and a disk hot-plug bug kept Managed Kubernetes control planes down for hours after power returned
    Nebius2026
  2. IL-0489 Rejoining a failed node of Magic Pages' three-node MySQL cluster broke its metadata and took customer websites down
    Magic Pages2025
  3. IL-0039 New Kubernetes worker nodes could not join any cluster because a package URL hardcoded in kops nodeup broke
    FREE NOW2019

Configuration change13 cases

Component
Kubernetes API server4Cloud load balancer3Admission webhook2Client-side DNS resolver and cache2kubelet2Argo CD1CDN and edge network1cert-manager1Cilium1Cluster dashboard1CNI plugin1Configuration templating1conntrack table1CoreDNS1Authoritative DNS servers and records1etcd1external-dns1GKE control plane1HTTP and RPC client library1Ingress controller1Node launch template1Linux kernel1Container memory limit1Persistent volumes and stateful sets1Prometheus1RBAC and service accounts1Container registry and its credentials1Telemetry agent1Terraform1
Contributing factor
Monitoring blind spot13Excessive blast radius3Unrepresentative staging3Automation amplification2Unreviewed defaults2Change with a delayed effect2No configuration tests2Missed in review2Signal buried in noise1Unverified assumption1Ignored early signal1Cache-masked failure1Unknown limit1Limited debug access1No alert on limit1No health probes1No resource requests or limits1No runbook1No safeguard on destructive change1No staged rollout1Retry storm1
Organisation
DFDS3OpenAI2Aqua Security1Datadog1GitHub1InfluxData1Jetstack1JW Player1MIT Open Learning1NU.nl1
  1. IL-0512 xpro.mit.edu returned 503 for an hour when its Kubernetes/APISIX origin certificate expired after renewal failed
    MIT Open Learning2026
  2. IL-0200 Newly provisioned Kubernetes nodes ran an incompatible etcd version and broke cluster consensus in several GitHub production clusters, causing 5XX errors on Git operations and page loads
    GitHub2026
  3. IL-0005 A new telemetry service overloaded the Kubernetes API servers of large clusters and broke DNS-based service discovery for all OpenAI services
    OpenAI2024
  4. IL-0068 Anonymous access to the Kubernetes API server used to plant a ClusterRoleBinding backdoor and a cryptominer DaemonSet
    Aqua Security2023
  5. IL-0116 A CNI configuration change incompatible with a hardware variant that exists only in production cut network connectivity in an OpenAI GPU cluster and took text-davinci-003 fully down
    OpenAI2023
  6. IL-0009 A duplicated generated name made Argo CD replace the InfluxDB Cloud production Application and delete the whole production environment
    InfluxData2022
  7. IL-0461 Every rollout of Datadog's metrics query service on Kubernetes raised errors that the logs blamed on DNS
    Datadog2022
  8. IL-0111 Switching the main Kubernetes ingress controller from Traefik v1 to v2 stopped X-Forwarded headers being passed, which broke the Okta login flow for every service using Okta and stopped DFDS online bookings for over an hour
    DFDS2021
  9. IL-0108 An Alcide admission controller applied to the production cluster during a workshop made create and update requests on Deployments time out
    DFDS2020
  10. IL-0365 A Terraform change to auto-scaling groups detached all worker nodes from both ALBs and cut off every cluster service
    DFDS2020
  11. IL-0008 An unresponsive OPA validating webhook crash-looped the API server during a GKE master upgrade, and node auto-repair then recreated every node
    Jetstack2019
  12. IL-0052 A cryptocurrency miner ran on every node of the development and staging clusters through a public, unauthenticated Weave Scope dashboard
    JW Player2019
  13. IL-0035 Worker nodes went NotReady after a new ElastAlert rule without memory limits used up node memory
    NU.nl2019

Breaking change in an upgrade5 cases

Component
Admission webhook3Ingress controller3Cloud load balancer2Helm2Calico1CDN and edge network1cert-manager1ConfigMaps and Secrets1CoreDNS1Client-side DNS resolver and cache1etcd1Kubernetes API server1kube-proxy1kubelet1
Contributing factor
Monitoring blind spot5Change with a delayed effect2Harmful manual action2Unrepresentative staging2Unverified assumption1Untested backup1Configuration not in code1Cache-masked failure1Partial rollout1Ineffective rollback1Single point of failure1
Organisation
CircleCI1Honeycomb1Jenkins project1PortSwigger1Reddit1
  1. IL-0405 An expired AWS Load Balancer Controller webhook certificate left no healthy targets and took BSEE SaaS down
    PortSwigger2026
  2. IL-0203 A routine Kubernetes cluster system update made the AWS Load Balancer Controller fail, so new pods were never registered with the ALB and Honeycomb's EU1 ingest and UI went down once the last old pods were replaced
    Honeycomb2024
  3. IL-0060 Kubernetes 1.24 upgrade removed the master node label that Calico route reflectors selected on, dropping all pod routes and taking Reddit down for 314 minutes
    Reddit2023
  4. IL-0230 kube-proxy sync failures during a Kubernetes upgrade corrupted node iptables, and CircleCI jobs stopped starting
    CircleCI2023
  5. IL-0522 An ingress-nginx chart upgrade deleted the jenkins.io Ingress and Fastly served HTTP/503 errors for hours
    Jenkins project2021

Faulty deployment2 cases

Component
Ingress controller2Calico1Cloud load balancer1Pod termination1
Contributing factor
Monitoring blind spot2Unverified assumption1Excessive blast radius1Unreviewed defaults1No runbook1
Organisation
CircleCI1Yonghyun Kim1
  1. IL-0188 Rolling updates on EKS returned 502 errors for about 30 seconds because Kubernetes deleted old pods as soon as new pods passed readiness, before the ALB had marked the new pod targets healthy
    Yonghyun Kim2025
  2. IL-0086 A deployment meant for a CircleCI test Kubernetes cluster also removed Traefik and Calico resources from the production cluster, taking down the CircleCI UI and runner.circleci.com
    CircleCI2021

Software defect8 cases

Component
kubelet3Linux kernel3Kubernetes API server2Admission webhook1BGP routing1Container CPU limit1Elasticsearch or OpenSearch1etcd1GKE control plane1GPU driver1Horizontal pod autoscaler1Node launch template1Linkerd1Network control plane (SDN)1Node operating system image1Telemetry agent1Terraform1AWS VPC CNI1
Contributing factor
Monitoring blind spot8Change with a delayed effect2Ignored early signal2Unpatched known bug2Overlapping failures2Ineffective rollback2Single point of failure2Automation amplification1Unknown limit1Harmful manual action1No health probes1No staged rollout1Gap in automated tests1
Organisation
Airbnb1Barush Mendez1Binder Project1DFDS1Monzo1Nebius1Omio1PrometheusKube1
  1. IL-0233 A VPC control plane release broke IP alias routes in Nebius eu-west1, and Managed Kubernetes pods lost connectivity
    Nebius2025
  2. IL-0138 A failed Nvidia driver build left broken kernel packages on the only Kubernetes master node, which hit a kernel panic on reboot after a power outage and took the homelab cluster down for six days
    Barush Mendez2025
  3. IL-0112 The daily node rollover put DFDS's EKS nodes on an AMI later recalled for a kernel issue, causing "PLEG is not healthy" NodeNotReady flaps and pod network timeouts
    DFDS2022
  4. IL-0403 mybinder.org could not create pods for hours as the GKE gke-resource-quotas quota wrongly reported 15k pods in use
    Binder Project2022
  5. IL-0032 Mutating admission controller generated broken JSON patches, silently blocking deploys for a week until a production service ran down to two pods
    Airbnb2020
  6. IL-0041 fluent-bit silently stopped shipping application logs from one node to Elasticsearch
    PrometheusKube2020
  7. IL-0012 CPU limits and an unpatched kernel CFS quota bug throttled multi-threaded services into stalls, failed health checks and timeouts
    Omio2020
  8. IL-0003 linkerd lost Kubernetes service discovery after an etcd reconfiguration, and restarting every linkerd instance turned failed payments into a full platform outage
    Monzo2017

Certificate or credential expiry3 cases

Component
Admission webhook1cert-manager1Cluster cloud identity1CNI plugin1Node pool1Persistent volumes and stateful sets1Policy engine1Secrets store1
Contributing factor
Monitoring blind spot3Unverified assumption1Unreviewed defaults1Ignored early signal1Harmful manual action1Single point of failure1Deferred upgrades1
Organisation
DFDS1GoCardless1Jenkins project1
  1. IL-0107 Kyverno stuck in a crash loop behind a webhook with failurePolicy fail blocked every change to Kubernetes Service objects for 88 minutes
    DFDS2022
  2. IL-0521 AKS pods could not mount volumes and new nodes got no CNI network interface after the cluster service principal expired
    Jenkins project2021
  3. IL-0460 Vault served an expired TLS certificate and almost every GoCardless service pod crash-looped, a 22-minute full outage
    GoCardless2020

Cloud provider outage1 case

Component
Cluster autoscalerNode pool
Contributing factor
Limited debug accessMonitoring blind spotOverlapping failures
Organisation
GitLab
  1. IL-0205 The GKE cluster autoscaler could not add nodes to GitLab.com's production cluster because Google Cloud ran out of SSD persistent disks in us-east1
    GitLab2021

unknown1 case

Component
etcdKubernetes API serverkubelet
Contributing factor
Unreviewed defaultsIgnored early signalHarmful manual actionMonitoring blind spotSingle point of failure
Organisation
NRE Labs
  1. IL-0059 Single-master Kubernetes control plane kept dying because the kubelet restarted a healthy etcd on a failing kubeadm liveness probe
    NRE Labs2018