Incident Library
  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases23 cases

23 to 23 of 23

  1. IL-0040 Parallel Helm installs saturated small gp2 root volumes on a kops cluster and left all three workers NotReady with the kubelet or Docker down
    Saltside2017
    Symptoms

    While running the CI process on a local machine to test an application Helm chart, the SRE lead watched the pods with watch kubectl get pods. All pods reached Running, and then some went back to Pending. Two hours later the problem had grown and some pods showed the phase Unknown.

    Pods PendingImage pull failuresNode NotReadyService outage
    Trigger

    A CI run installed the application's Helm charts for all four markets (Efritin.com, Tonaton.com, Ikman.lk and Bikroy.com) on the nonproduction cluster at the same time.

    Root cause
    Capacity exhaustion
    Component
    Node root diskkubeletContainer runtimeHelm
    Contributing factor
    Unreviewed defaultsMonitoring blind spotIgnored early signalSingle point of failure
    Detection

    The SRE lead saw pods go from Running back to Pending while watching kubectl get pods during the CI run; no monitoring raised an alert, even when all three workers were not ready.

    Remediation
    • terminated the three worker nodes and let the auto scaling group replace them
    Action items
    • Fix the Datadog integration for the nonproduction AWS account and repeat the installs with telemetry
    • Increase the worker root EBS volumes from 20GB to 120GB
    • Keep common base images pulled on every node with a DaemonSet until kops supports user hooks
    • Configure different allocation limits for the different layers in Kubernetes
    • Discuss a highly available Tiller with the Helm team
    • Find a way to throttle the number of newly created pods
    Lessons

    An ordinary deploy of the application could collapse the cluster, because the default 20GB gp2 root volumes could not keep up with parallel image pulls and container starts. Without node metrics and pod phase metrics, three NotReady workers raised no alert. A single Tiller replica can block the deploy that would fix an outage.

    View case
Check your cluster for this

These 23 cases involve kubelet and Kubernetes API server. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause8

Capacity exhaustion7 cases

Component
kubelet7Node root disk3CI system2Container runtime2Container memory limit2Cloud block storage1CoreDNS1Credential broker for pods1Client-side DNS resolver and cache1Helm1Kubernetes API server1Linux kernel1Node pool1Prometheus1
Contributing factor
Unreviewed defaults4Single point of failure4Monitoring blind spot3Ignored early signal2Unknown limit1No caching1No capacity planning1No resource requests or limits1Retry storm1
Organisation
Backpackr1Kubernetes project1LINE1Onkar Raghunath Shelke1Saltside1Scaleway1Zalando1
  1. IL-0071 Worker node running out of inodes while disk space looked fine, because containerd kept a full node_modules tree in each of several image layer snapshots
    Onkar Raghunath Shelke2026
  2. IL-0027 All CoreDNS pods landed on one node, the node ran out of memory, and DNS for the whole service went down
    Backpackr2025
  3. IL-0169 The Scaleway Kubernetes API in FR-PAR ran out of memory under simultaneous deployments, and because it also handled node authentication, Kapsule and Kosmos nodes went NotReady
    Scaleway2024
  4. IL-0176 Prometheus in a 235-node Kubernetes cluster crash-looped because over two million series exhausted host memory and the kernel OOM killer chose it first, as Burstable QoS gave it oom_score_adj 999
    LINE2020
  5. IL-0002 CI builder pods could not get AWS IAM credentials because kubelet fell minutes behind its API rate limit
    Zalando2019
  6. IL-0401 Prow jobs on the k8s-prow-build GKE cluster failed and merges stalled as nodes hit IOPS throttling and kernel panics
    Kubernetes project2019
  7. IL-0040 Parallel Helm installs saturated small gp2 root volumes on a kops cluster and left all three workers NotReady with the kubelet or Docker down
    Saltside2017

Certificate or credential expiry2 cases

Component
Kubernetes API server2kubelet2Calico1etcd1
Contributing factor
Unreviewed defaults1Unknown limit1No alert on limit1
Organisation
ckck8031dbhang1
  1. IL-0183 kubeadm's one-year leaf certificates expired, so kube-apiserver could not reach etcd, restarted in a loop and left the cluster unreachable
    ckck8032026
  2. IL-0184 JupyterHub's KubeSpawner could not create containers and kubectl was refused because the API server and other internal certificates of a kubeadm cluster had expired
    dbhang2021

DNS or network fault1 case

Component
Kubernetes API serverkubelet
Contributing factor
–
Organisation
DigitalOcean
  1. IL-0154 Line cards on an SFO3 core router rebooted together and halved backbone capacity, and DOKS nodes whose guest networking did not recover stayed NotReady with degraded API servers until all affected nodes in the SFO3 DOKS fleet were recycled
    DigitalOcean2024

Configuration change3 cases

Component
kubelet3Admission webhook1GKE control plane1Kubernetes API server1Linux kernel1Container memory limit1Node operating system image1Prometheus1
Contributing factor
Excessive blast radius2Unreviewed defaults2Monitoring blind spot2Automation amplification1Change with a delayed effect1Limited debug access1No alert on limit1No health probes1No resource requests or limits1
Organisation
DigitalOcean1Jetstack1NU.nl1
  1. IL-0153 DOKS worker nodes went NotReady at 06:00 UTC on three days because a systemd timer in the Debian image ran apt upgrade and killed it mid kernel upgrade, breaking node networking
    DigitalOcean2023
  2. IL-0008 An unresponsive OPA validating webhook crash-looped the API server during a GKE master upgrade, and node auto-repair then recreated every node
    Jetstack2019
  3. IL-0035 Worker nodes went NotReady after a new ElastAlert rule without memory limits used up node memory
    NU.nl2019

Breaking change in an upgrade2 cases

Component
Client-side DNS resolver and cache2kubelet2CoreDNS1kube-proxy1Node operating system image1Container registry and its credentials1
Contributing factor
Automation amplification1Excessive blast radius1Harmful manual action1Monitoring blind spot1No staged rollout1Partial rollout1Ineffective rollback1
Organisation
CircleCI1Microsoft Azure1
  1. IL-0230 kube-proxy sync failures during a Kubernetes upgrade corrupted node iptables, and CircleCI jobs stopped starting
    CircleCI2023
  2. IL-0144 A systemd security update installed by unattended-upgrades lost the DNS configuration of Ubuntu 18.04 VMs, so AKS kubelets could not resolve registry names and pods failed with ImagePullBackOff
    Microsoft Azure2022

Software defect6 cases

Component
kubelet6Linux kernel2Calico1Container CPU limit1etcd1Kubernetes API server1Kubernetes controller manager1Node launch template1Linkerd1Node operating system image1Pod termination1Terraform1AWS VPC CNI1
Contributing factor
Monitoring blind spot3Unpatched known bug2No health probes2Overlapping failures2Automation amplification1Excessive blast radius1Unknown limit1Harmful manual action1No staged rollout1Ineffective rollback1Single point of failure1
Organisation
AirMap1DFDS1Google Cloud1Monzo1Oath1Omio1
  1. IL-0112 The daily node rollover put DFDS's EKS nodes on an AMI later recalled for a kernel issue, causing "PLEG is not healthy" NodeNotReady flaps and pod network timeouts
    DFDS2022
  2. IL-0139 Pods on GKE clusters with Calico network policy got stuck Terminating or Pending after upgrading to 1.22 or later because of a race condition in the Calico component
    Google Cloud2022
  3. IL-0012 CPU limits and an unpatched kernel CFS quota bug throttled multi-threaded services into stalls, failed health checks and timeouts
    Omio2020
  4. IL-0074 All AirMap platform services down as worker nodes went NotReady and pods crash-looped on network and DNS failures
    AirMap2018
  5. IL-0038 kubelet rollout hit a Kubernetes bug and left every node in a cluster NotReady, triggering mass pod eviction
    Oath2018
  6. IL-0003 linkerd lost Kubernetes service discovery after an etcd reconfiguration, and restarting every linkerd instance turned failed payments into a full platform outage
    Monzo2017

Cloud provider outage1 case

Component
AKS control planeKubernetes API serverkubelet
Contributing factor
Limited debug access
Organisation
UC Berkeley
  1. IL-0469 The managed Azure Kubernetes API server for the data100 hub stopped answering, so the stopped hub pod was not recreated
    UC Berkeley2019

unknown1 case

Component
etcdKubernetes API serverkubelet
Contributing factor
Unreviewed defaultsIgnored early signalHarmful manual actionMonitoring blind spotSingle point of failure
Organisation
NRE Labs
  1. IL-0059 Single-master Kubernetes control plane kept dying because the kubelet restarted a healthy etcd on a failing kubeadm liveness probe
    NRE Labs2018