Incident Library
  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases53 cases

53 to 53 of 53

  1. IL-0040 Parallel Helm installs saturated small gp2 root volumes on a kops cluster and left all three workers NotReady with the kubelet or Docker down
    Saltside2017
    Symptoms

    While running the CI process on a local machine to test an application Helm chart, the SRE lead watched the pods with watch kubectl get pods. All pods reached Running, and then some went back to Pending. Two hours later the problem had grown and some pods showed the phase Unknown.

    Pods PendingImage pull failuresNode NotReadyService outage
    Trigger

    A CI run installed the application's Helm charts for all four markets (Efritin.com, Tonaton.com, Ikman.lk and Bikroy.com) on the nonproduction cluster at the same time.

    Root cause
    Capacity exhaustion
    Component
    Node root diskkubeletContainer runtimeHelm
    Contributing factor
    Unreviewed defaultsMonitoring blind spotIgnored early signalSingle point of failure
    Detection

    The SRE lead saw pods go from Running back to Pending while watching kubectl get pods during the CI run; no monitoring raised an alert, even when all three workers were not ready.

    Remediation
    • terminated the three worker nodes and let the auto scaling group replace them
    Action items
    • Fix the Datadog integration for the nonproduction AWS account and repeat the installs with telemetry
    • Increase the worker root EBS volumes from 20GB to 120GB
    • Keep common base images pulled on every node with a DaemonSet until kops supports user hooks
    • Configure different allocation limits for the different layers in Kubernetes
    • Discuss a highly available Tiller with the Helm team
    • Find a way to throttle the number of newly created pods
    Lessons

    An ordinary deploy of the application could collapse the cluster, because the default 20GB gp2 root volumes could not keep up with parallel image pulls and container starts. Without node metrics and pod phase metrics, three NotReady workers raised no alert. A single Tiller replica can block the deploy that would fix an outage.

    View case
Check your cluster for this

These 53 cases involve Client-side DNS resolver and cache and CoreDNS. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause10

Capacity exhaustion23 cases

Component
Client-side DNS resolver and cache7CoreDNS6Kubernetes API server5kubelet4etcd3VPC subnet or pod address range3Cluster autoscaler2CNI plugin2conntrack table2Container runtime2Container CPU limit2Database2kube-proxy2Linux kernel2Container memory limit2Node root disk2Node pool2Prometheus2AWS VPC CNI2VPC DNS resolver2In-memory cache1CI system1Cilium1Credential broker for pods1Helm1Horizontal pod autoscaler1Ingress controller1Istio1Jobs and CronJobs1Karpenter1kube-dns1Language runtime settings1Cluster authentication provider1Pod termination1Telemetry agent1
Contributing factor
Unreviewed defaults23Unknown limit7Monitoring blind spot5Single point of failure4Harmful manual action3Retry storm3Unverified assumption2Ignored early signal2No caching2No capacity planning2Unrepresentative staging2Excessive blast radius1Change with a delayed effect1Limited debug access1No resource requests or limits1Overlapping failures1Ineffective rollback1
Organisation
Backpackr2Palark2Zalando2Adevinta1Airbnb1Chatwork1CyberAgent1Cybozu1Fullstaq1incident.io1loveholidays1MonotaRO1Neon1Onkar Raghunath Shelke1Render1Saltside1Temporal1Tinder1UK Ministry of Justice1unknown1
  1. IL-0071 Worker node running out of inodes while disk space looked fine, because containerd kept a full node_modules tree in each of several image layer snapshots
    Onkar Raghunath Shelke2026
  2. IL-0435 Error rates rose as Aurora MySQL killed connections and declined queries to avoid OOM while load metrics stayed flat
    CyberAgent2026
  3. IL-0048 DNS lookups timed out in a large EKS cluster when CoreDNS hit the 1024 packets-per-second link-local limit, driven by sudo hostname lookups
    unknown2025
  4. IL-0006 VPC subnets ran out of IP addresses because the AWS VPC CNI kept IPs on full nodes, and WARM_IP_TARGET changes then blocked pod starts after every pod deletion
    Neon2025
  5. IL-0170 During a Lunar New Year event, traffic more than three times normal filled the nf_conntrack table on the EKS node running the Istio ingress gateway, dropping packets and taking down Backpackr's web and app services
    Backpackr2025
  6. IL-0027 All CoreDNS pods landed on one node, the node ran out of memory, and DNS for the whole service went down
    Backpackr2025
  7. IL-0204 An etcd memory spike overloaded the control plane of a Render Frankfurt cluster, and DNS then failed because CoreDNS ran on the rebooted control plane nodes and could not start without the API server
    Render2024
  8. IL-0007 The AWS VPC CNI could not assign IP addresses to pods as an EKS cluster ran out of IPv4 space
    Adevinta2024
  9. IL-0440 Pods on Cybozu's Neco Kubernetes platform got frequent no such host and i/o timeout DNS errors from dropped packets
    Cybozu2024
  10. IL-0046 Prometheus restarted again and again because its WAL replay took longer than the startup probe allowed
    UK Ministry of Justice2024
  11. IL-0386 After a move to Google Cloud, Postgres connection timeouts and memcached i/o timeouts spiked on GKE nodes
    incident.io2023
  12. IL-0433 Shifting API traffic from EC2 to EKS on Fargate overloaded two CoreDNS pods and caused mass name-resolution errors
    MonotaRO2023
  13. IL-0028 Temporal History service latency rose because its pods were CPU-throttled while CPU usage read below 60 percent
    Temporal2023
  14. IL-0195 The kube-apiservers of a cluster with more than 200 nodes ran out of memory and crashed one after another under mass LIST requests from Cilium agents
    Palark2023
  15. IL-0194 After a kube-apiserver restart, pod LIST requests from every pod of a Vector DaemonSet nearly tripled etcd memory and restarted all master nodes
    Palark2023
  16. IL-0445 Chatwork's service on EKS became unstable as conntrack tables on some nodes overflowed and connections were dropped
    Chatwork2021
  17. IL-0015 A GKE cluster's /16 pod address range ran out at 256 nodes, leaving new pods Pending and node scale-ups failing
    loveholidays2020
  18. IL-0033 Uneven load across kube-dns pods and a DNS packets-per-second limit pushed service latency to 20 seconds
    Airbnb2019
  19. IL-0002 CI builder pods could not get AWS IAM credentials because kubelet fell minutes behind its API rate limit
    Zalando2019
  20. IL-0076 Kubernetes masters went down after Dex filled etcd with AuthRequest objects and deleting its namespace started an endless list-and-timeout loop
    Fullstaq2019
  21. IL-0472 Node-to-pod traffic and DNS failed on Tinder's 605-node Flannel cluster when every node's ARP cache passed gc_thresh3
    Tinder2019
  22. IL-0001 Cluster-wide DNS outage after CoreDNS pods were OOM-killed again and again under a burst of DNS queries
    Zalando2019
  23. IL-0040 Parallel Helm installs saturated small gp2 root volumes on a kops cluster and left all three workers NotReady with the kubelet or Docker down
    Saltside2017

DNS or network fault6 cases

Component
conntrack table3Client-side DNS resolver and cache2kube-proxy2Linux kernel2AWS VPC CNI2Cloud load balancer1CoreDNS1Horizontal pod autoscaler1Ingress controller1Node operating system image1Pod termination1Telemetry agent1
Contributing factor
Unreviewed defaults6Unverified assumption2Unknown limit2Ignored early signal1Unpatched known bug1
Organisation
MindTickle2Adevinta1Lotte E-commerce1Prezi1Uzabase1
  1. IL-0448 Requests from Uzabase's Kubernetes Job pods to an LLM agent randomly never got responses when they ran past 15 minutes
    Uzabase2026
  2. IL-0083 Intermittent 5xx errors in a multi-tenant Kubernetes cluster came from internal DNS: the node-local dnsmasq hit its 150 concurrent query limit, did not cache cluster.local answers, and ndots search lookups flooded CoreDNS
    Adevinta2023
  3. IL-0178 On EKS, HPA scale-in returned 502 errors because containers stopped before their pod IPs left the Service endpoints, and after blue/green switches conntrack kept sending established connections to deleted pods behind an instance-mode ALB
    Lotte E-commerce2022
  4. IL-0056 Calls from EKS pods to services behind an NLB in another VPC failed intermittently because the AWS VPC CNI did source NAT on the node
    MindTickle2020
  5. IL-0057 Higher p99 latency after moving a reverse proxy to EKS, from outgoing connections delayed by a SNAT conntrack race
    Prezi2019
  6. IL-0047 Pod-to-pod requests delayed by 5, 10 or 15 seconds because of DNS lookups lost to a conntrack race
    MindTickle2019

Configuration change8 cases

Component
kubelet2Container memory limit2Node pool2Cilium1Cloud load balancer1Cluster dashboard1Configuration templating1Container runtime1Credential broker for pods1Jobs and CronJobs1Node launch template1Linux kernel1Node operating system image1Prometheus1Service discovery1Telemetry agent1Terraform1
Contributing factor
Unreviewed defaults8Automation amplification2Excessive blast radius2Change with a delayed effect2Ignored early signal2Monitoring blind spot2No alert on limit2Unrepresentative staging2Limited debug access1Harmful manual action1No configuration tests1No resource requests or limits1No runbook1Retry storm1
Organisation
Adevinta1Adyen1DFDS1DigitalOcean1Grafana Labs1JW Player1NU.nl1Target1
  1. IL-0246 New pods on Adyen's big data Kubernetes nodes could not start, failing with "Cilium API client timeout exceeded"
    Adyen2026
  2. IL-0153 DOKS worker nodes went NotReady at 06:00 UTC on three days because a systemd timer in the Debian image ran apt upgrade and killed it mid kernel upgrade, breaking node networking
    DigitalOcean2023
  3. IL-0109 A retried node rollover on a DFDS EKS cluster set the Auto Scaling group back to 15 instances, and the default termination policy removed every healthy worker node, leaving the cluster with no nodes
    DFDS2023
  4. IL-0191 A label selector accidentally added to the Loki distributor Service matched only the new pods of a rollout, which took all write traffic, were OOM-killed and stalled the rollout
    Grafana Labs2022
  5. IL-0189 A JVM service moved from EC2 to Kubernetes answered up to ten times slower because KIAM's 15-minute credentials made the AWS Java SDK fetch new credentials on every request
    Adevinta2019
  6. IL-0052 A cryptocurrency miner ran on every node of the development and staging clusters through a public, unauthenticated Weave Scope dashboard
    JW Player2019
  7. IL-0035 Worker nodes went NotReady after a new ElastAlert rule without memory limits used up node memory
    NU.nl2019
  8. IL-0055 A failing CronJob with the default concurrency policy piled up thousands of restarting pods and took down a cluster
    Target2018

Certificate or credential expiry2 cases

Component
Admission webhook1Calico1etcd1Kubernetes API server1kubelet1Policy engine1
Contributing factor
Unreviewed defaults2Ignored early signal1Monitoring blind spot1Deferred upgrades1
Organisation
ckck8031DFDS1
  1. IL-0183 kubeadm's one-year leaf certificates expired, so kube-apiserver could not reach etcd, restarted in a loop and left the cluster unreachable
    ckck8032026
  2. IL-0107 Kyverno stuck in a crash loop behind a webhook with failurePolicy fail blocked every change to Kubernetes Service objects for 88 minutes
    DFDS2022

Faulty deployment2 cases

Component
Cloud load balancer1Cluster autoscaler1Descheduler1Ingress controller1Node pool1Pod termination1
Contributing factor
Unreviewed defaults2Unverified assumption1Monitoring blind spot1No alert on limit1No headroom1
Organisation
UK Ministry of Justice1Yonghyun Kim1
  1. IL-0188 Rolling updates on EKS returned 502 errors for about 30 seconds because Kubernetes deleted old pods as soon as new pods passed readiness, before the ALB had marked the new pod targets healthy
    Yonghyun Kim2025
  2. IL-0020 During an EKS 1.30 upgrade the descheduler tried to reschedule every pod on the cordoned nodes at once and production workloads went down
    UK Ministry of Justice2025

Dependency failure2 cases

Component
Admission webhook1Liveness and readiness probes1Istio1Node operating system image1
Contributing factor
Unreviewed defaults2Automation amplification1Excessive blast radius1Unknown limit1
Organisation
Tetrate (unnamed U.S. federal customer)1Yonghyun Kim1
  1. IL-0187 A brief Spring Cloud Config Server downtime failed every client application's liveness probe, because the actuator health check included the config server, and Kubernetes restarted the pods in a kill loop
    Yonghyun Kim2025
  2. IL-0103 During an EKS node AMI update, Istio sidecars crash-looped because hundreds of pods restarting at once overloaded the external certificate authority behind istiod
    Tetrate (unnamed U.S. federal customer)2025

Software defect7 cases

Component
etcd2Ingress controller2Admission webhook1Argo CD1conntrack table1CoreDNS1Client-side DNS resolver and cache1Istio1Kubernetes API server1kube-proxy1Linux kernel1Node pool1Persistent volumes and stateful sets1Telemetry agent1
Contributing factor
Unreviewed defaults7Excessive blast radius2Unverified assumption1Credential in configuration1No safeguard on destructive change1
Organisation
Cisco ThousandEyes1Cycode1Grafana Labs1Nordstrom1Pinterest1Toyota Connected1Wiz1
  1. IL-0104 After a CoreDNS pod was OOMKilled, the talk names three causes: Istio's aggressive DNS cache refresh on one reused client port, kube-proxy not cleaning up conntrack entries of terminated pods, and the 2-minute UDP conntrack timeout
    Cisco ThousandEyes2025
  2. IL-0247 One in a million search requests took 100 times longer on Pinterest's Manas leaf nodes after moving to Kubernetes
    Pinterest2025
  3. IL-0053 Unauthenticated ingress-nginx admission webhook let anyone on the pod network run code and read every Secret in the cluster (IngressNightmare, CVE-2025-1974)
    Wiz2025
  4. IL-0065 Any pod could take over the cluster by rewriting Argo CD's unauthenticated Redis cache (CVE-2024-31989)
    Cycode2024
  5. IL-0190 Cortex distributors kept a stuck connection to a dead etcd leader after a Kubernetes upgrade terminated its node, failing writes to Hosted Prometheus for 12 minutes
    Grafana Labs2020
  6. IL-0050 Pods on Kubernetes failed to resolve an external Redis hostname because of ndots:5, the musl resolver in Alpine images, and Netty's DNS resolver
    Toyota Connected2018
  7. IL-0072 High etcd write latency made the API servers return stale data, and the production cluster flipped between two views of its state
    Nordstrom2017

Breaking change in an upgrade1 case

Component
CiliumNode operating system imagesystemd-networkd
Contributing factor
Automation amplificationUnreviewed defaultsChange with a delayed effectNo staged rollout
Organisation
Datadog
  1. IL-0198 An automatic Ubuntu security update restarted systemd-networkd, which flushed the IP rules Cilium relied on and cut more than 60 percent of Datadog's Kubernetes nodes off the network across regions and cloud providers
    Datadog2023

Operator error1 case

Component
Ingress controllerkube-proxyNode pool
Contributing factor
Unverified assumptionUnreviewed defaults
Organisation
DevOps Hof
  1. IL-0054 All ingress traffic lost while draining a GKE node pool because the load balancer only targeted new nodes without ingress-nginx pods
    DevOps Hof2019

unknown1 case

Component
etcdKubernetes API serverkubelet
Contributing factor
Unreviewed defaultsIgnored early signalHarmful manual actionMonitoring blind spotSingle point of failure
Organisation
NRE Labs
  1. IL-0059 Single-master Kubernetes control plane kept dying because the kubelet restarted a healthy etcd on a failing kubeadm liveness probe
    NRE Labs2018