Incident Library
Applied filters ComponentkubeletFactorMonitoring blind spotClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases8 cases

8 to 8 of 8

  1. IL-0040 Parallel Helm installs saturated small gp2 root volumes on a kops cluster and left all three workers NotReady with the kubelet or Docker down
    Saltside2017
    Symptoms

    While running the CI process on a local machine to test an application Helm chart, the SRE lead watched the pods with watch kubectl get pods. All pods reached Running, and then some went back to Pending. Two hours later the problem had grown and some pods showed the phase Unknown.

    Pods PendingImage pull failuresNode NotReadyService outage
    Trigger

    A CI run installed the application's Helm charts for all four markets (Efritin.com, Tonaton.com, Ikman.lk and Bikroy.com) on the nonproduction cluster at the same time.

    Root cause
    Capacity exhaustion
    Component
    Node root diskkubeletContainer runtimeHelm
    Contributing factor
    Unreviewed defaultsMonitoring blind spotIgnored early signalSingle point of failure
    Detection

    The SRE lead saw pods go from Running back to Pending while watching kubectl get pods during the CI run; no monitoring raised an alert, even when all three workers were not ready.

    Remediation
    • terminated the three worker nodes and let the auto scaling group replace them
    Action items
    • Fix the Datadog integration for the nonproduction AWS account and repeat the installs with telemetry
    • Increase the worker root EBS volumes from 20GB to 120GB
    • Keep common base images pulled on every node with a DaemonSet until kops supports user hooks
    • Configure different allocation limits for the different layers in Kubernetes
    • Discuss a highly available Tiller with the Helm team
    • Find a way to throttle the number of newly created pods
    Lessons

    An ordinary deploy of the application could collapse the cluster, because the default 20GB gp2 root volumes could not keep up with parallel image pulls and container starts. Without node metrics and pod phase metrics, three NotReady workers raised no alert. A single Tiller replica can block the deploy that would fix an outage.

    View case
Check your cluster for this

These 8 cases involve kubelet and Kubernetes API server. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause4

Capacity exhaustion2 cases

Component
Container runtime2kubelet2Node root disk2Helm1
Contributing factor
Unreviewed defaults2Monitoring blind spot2Ignored early signal1Unknown limit1Single point of failure1
Organisation
Onkar Raghunath Shelke1Saltside1
  1. IL-0071 Worker node running out of inodes while disk space looked fine, because containerd kept a full node_modules tree in each of several image layer snapshots
    Onkar Raghunath Shelke2026
  2. IL-0040 Parallel Helm installs saturated small gp2 root volumes on a kops cluster and left all three workers NotReady with the kubelet or Docker down
    Saltside2017

Software defect3 cases

Component
kubelet3Linux kernel2Container CPU limit1etcd1Kubernetes API server1Node launch template1Linkerd1Node operating system image1Terraform1AWS VPC CNI1
Contributing factor
Monitoring blind spot3Overlapping failures2Automation amplification1Unpatched known bug1Unknown limit1Harmful manual action1No health probes1No staged rollout1Ineffective rollback1Single point of failure1
Organisation
DFDS1Monzo1Omio1
  1. IL-0112 The daily node rollover put DFDS's EKS nodes on an AMI later recalled for a kernel issue, causing "PLEG is not healthy" NodeNotReady flaps and pod network timeouts
    DFDS2022
  2. IL-0012 CPU limits and an unpatched kernel CFS quota bug throttled multi-threaded services into stalls, failed health checks and timeouts
    Omio2020
  3. IL-0003 linkerd lost Kubernetes service discovery after an etcd reconfiguration, and restarting every linkerd instance turned failed payments into a full platform outage
    Monzo2017

Configuration change2 cases

Component
kubelet2Admission webhook1GKE control plane1Kubernetes API server1Container memory limit1Prometheus1
Contributing factor
Excessive blast radius2Monitoring blind spot2Automation amplification1Unreviewed defaults1No alert on limit1No health probes1No resource requests or limits1
Organisation
Jetstack1NU.nl1
  1. IL-0008 An unresponsive OPA validating webhook crash-looped the API server during a GKE master upgrade, and node auto-repair then recreated every node
    Jetstack2019
  2. IL-0035 Worker nodes went NotReady after a new ElastAlert rule without memory limits used up node memory
    NU.nl2019

unknown1 case

Component
etcdKubernetes API serverkubelet
Contributing factor
Unreviewed defaultsIgnored early signalHarmful manual actionMonitoring blind spotSingle point of failure
Organisation
NRE Labs
  1. IL-0059 Single-master Kubernetes control plane kept dying because the kubelet restarted a healthy etcd on a failing kubeadm liveness probe
    NRE Labs2018