Incident Library
Applied filters Root causeCapacity exhaustionFactorUnreviewed defaultsClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases17 cases

17 to 17 of 17

  1. IL-0040 Parallel Helm installs saturated small gp2 root volumes on a kops cluster and left all three workers NotReady with the kubelet or Docker down
    Saltside2017
    Symptoms

    While running the CI process on a local machine to test an application Helm chart, the SRE lead watched the pods with watch kubectl get pods. All pods reached Running, and then some went back to Pending. Two hours later the problem had grown and some pods showed the phase Unknown.

    Pods PendingImage pull failuresNode NotReadyService outage
    Trigger

    A CI run installed the application's Helm charts for all four markets (Efritin.com, Tonaton.com, Ikman.lk and Bikroy.com) on the nonproduction cluster at the same time.

    Root cause
    Capacity exhaustion
    Component
    Node root diskkubeletContainer runtimeHelm
    Contributing factor
    Unreviewed defaultsMonitoring blind spotIgnored early signalSingle point of failure
    Detection

    The SRE lead saw pods go from Running back to Pending while watching kubectl get pods during the CI run; no monitoring raised an alert, even when all three workers were not ready.

    Remediation
    • terminated the three worker nodes and let the auto scaling group replace them
    Action items
    • Fix the Datadog integration for the nonproduction AWS account and repeat the installs with telemetry
    • Increase the worker root EBS volumes from 20GB to 120GB
    • Keep common base images pulled on every node with a DaemonSet until kops supports user hooks
    • Configure different allocation limits for the different layers in Kubernetes
    • Discuss a highly available Tiller with the Helm team
    • Find a way to throttle the number of newly created pods
    Lessons

    An ordinary deploy of the application could collapse the cluster, because the default 20GB gp2 root volumes could not keep up with parallel image pulls and container starts. Without node metrics and pod phase metrics, three NotReady workers raised no alert. A single Tiller replica can block the deploy that would fix an outage.

    View case
Check your cluster for this

These 17 cases involve Kubernetes API server and CoreDNS. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Capacity exhaustion17 cases

Component
Kubernetes API server5CoreDNS4Client-side DNS resolver and cache4kubelet4etcd3VPC subnet or pod address range3Cluster autoscaler2Container runtime2Container memory limit2Node root disk2Node pool2Prometheus2AWS VPC CNI2CI system1Cilium1conntrack table1Container CPU limit1Credential broker for pods1Helm1Ingress controller1Istio1Karpenter1kube-dns1kube-proxy1Language runtime settings1Cluster authentication provider1Telemetry agent1VPC DNS resolver1
Contributing factor
Unreviewed defaults17Unknown limit6Monitoring blind spot5Single point of failure4Harmful manual action3Retry storm3Ignored early signal2No caching2No capacity planning2Unverified assumption1Excessive blast radius1Limited debug access1Ineffective rollback1
Organisation
Backpackr2Palark2Zalando2Adevinta1Airbnb1Fullstaq1loveholidays1Neon1Onkar Raghunath Shelke1Render1Saltside1Temporal1UK Ministry of Justice1unknown1
  1. IL-0071 Worker node running out of inodes while disk space looked fine, because containerd kept a full node_modules tree in each of several image layer snapshots
    Onkar Raghunath Shelke2026
  2. IL-0048 DNS lookups timed out in a large EKS cluster when CoreDNS hit the 1024 packets-per-second link-local limit, driven by sudo hostname lookups
    unknown2025
  3. IL-0006 VPC subnets ran out of IP addresses because the AWS VPC CNI kept IPs on full nodes, and WARM_IP_TARGET changes then blocked pod starts after every pod deletion
    Neon2025
  4. IL-0170 During a Lunar New Year event, traffic more than three times normal filled the nf_conntrack table on the EKS node running the Istio ingress gateway, dropping packets and taking down Backpackr's web and app services
    Backpackr2025
  5. IL-0027 All CoreDNS pods landed on one node, the node ran out of memory, and DNS for the whole service went down
    Backpackr2025
  6. IL-0204 An etcd memory spike overloaded the control plane of a Render Frankfurt cluster, and DNS then failed because CoreDNS ran on the rebooted control plane nodes and could not start without the API server
    Render2024
  7. IL-0007 The AWS VPC CNI could not assign IP addresses to pods as an EKS cluster ran out of IPv4 space
    Adevinta2024
  8. IL-0046 Prometheus restarted again and again because its WAL replay took longer than the startup probe allowed
    UK Ministry of Justice2024
  9. IL-0028 Temporal History service latency rose because its pods were CPU-throttled while CPU usage read below 60 percent
    Temporal2023
  10. IL-0195 The kube-apiservers of a cluster with more than 200 nodes ran out of memory and crashed one after another under mass LIST requests from Cilium agents
    Palark2023
  11. IL-0194 After a kube-apiserver restart, pod LIST requests from every pod of a Vector DaemonSet nearly tripled etcd memory and restarted all master nodes
    Palark2023
  12. IL-0015 A GKE cluster's /16 pod address range ran out at 256 nodes, leaving new pods Pending and node scale-ups failing
    loveholidays2020
  13. IL-0033 Uneven load across kube-dns pods and a DNS packets-per-second limit pushed service latency to 20 seconds
    Airbnb2019
  14. IL-0002 CI builder pods could not get AWS IAM credentials because kubelet fell minutes behind its API rate limit
    Zalando2019
  15. IL-0076 Kubernetes masters went down after Dex filled etcd with AuthRequest objects and deleting its namespace started an endless list-and-timeout loop
    Fullstaq2019
  16. IL-0001 Cluster-wide DNS outage after CoreDNS pods were OOM-killed again and again under a burst of DNS queries
    Zalando2019
  17. IL-0040 Parallel Helm installs saturated small gp2 root volumes on a kops cluster and left all three workers NotReady with the kubelet or Docker down
    Saltside2017