Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Component

kubelet

kubelet · kubelet

20 cases · 2017 to 2026

Root cause Capacity exhaustion6 Software defect6 Configuration change3 Certificate or credential expiry2 DNS or network fault1 unknown1 Breaking change in an upgrade1

Contributing factor Unreviewed defaults8 Monitoring blind spot8 Single point of failure6 Excessive blast radius4

Symptom Node NotReady8 Service outage7 Control plane unresponsive5 High error rate5

Organisation DigitalOcean2 AirMap1 Backpackr1 ckck8031 dbhang1

Cases, by severity

IL-0071 Worker node running out of inodes while disk space looked fine, because containerd kept a full node_modules tree in each of several image layer snapshotsOnkar Raghunath Shelke · 2026
Root cause Capacity exhaustion
IL-0183 kubeadm's one-year leaf certificates expired, so kube-apiserver could not reach etcd, restarted in a loop and left the cluster unreachableckck803 · 2026
Root cause Certificate or credential expiry
All CoreDNS pods landed on one node, the node ran out of memory, and DNS for the whole service went down

View on the graph

Automation amplification3

No health probes3

Ignored early signal2

Unpatched known bug2

Harmful manual action2

No alert on limit2

No resource requests or limits2

No staged rollout2

Overlapping failures2

Change with a delayed effect1

Limited debug access1

No capacity planning1

Ineffective rollback1

Intermittent connection failures4

Intermittent timeouts4

Monitoring outage4

Container crash loop3

DNS resolution failures3

Probe failures3

Image pull failures2

Control plane lag1

CPU throttling1

Credentials unavailable1

Deploys failing1

Node disk full1

Node recreation loop1

Pods stuck in Terminating1

Stale endpoint routing1

Microsoft Azure1

Onkar Raghunath Shelke1

IL-0027

Backpackr · 2025

Root cause Capacity exhaustion

IL-0154 Line cards on an SFO3 core router rebooted together and halved backbone capacity, and DOKS nodes whose guest networking did not recover stayed NotReady with degraded API servers until all affected nodes in the SFO3 DOKS fleet were recycledDigitalOcean · 2024

Root cause DNS or network fault

IL-0169 The Scaleway Kubernetes API in FR-PAR ran out of memory under simultaneous deployments, and because it also handled node authentication, Kapsule and Kosmos nodes went NotReadyScaleway · 2024

Root cause Capacity exhaustion

IL-0153 DOKS worker nodes went NotReady at 06:00 UTC on three days because a systemd timer in the Debian image ran apt upgrade and killed it mid kernel upgrade, breaking node networkingDigitalOcean · 2023

Root cause Configuration change

IL-0112 The daily node rollover put DFDS's EKS nodes on an AMI later recalled for a kernel issue, causing "PLEG is not healthy" NodeNotReady flaps and pod network timeoutsDFDS · 2022

Root cause Software defect

IL-0139 Pods on GKE clusters with Calico network policy got stuck Terminating or Pending after upgrading to 1.22 or later because of a race condition in the Calico componentGoogle Cloud · 2022

Root cause Software defect

IL-0144 A systemd security update installed by unattended-upgrades lost the DNS configuration of Ubuntu 18.04 VMs, so AKS kubelets could not resolve registry names and pods failed with ImagePullBackOffMicrosoft Azure · 2022

Root cause Breaking change in an upgrade

IL-0184 JupyterHub's KubeSpawner could not create containers and kubectl was refused because the API server and other internal certificates of a kubeadm cluster had expireddbhang · 2021

Root cause Certificate or credential expiry

IL-0176 Prometheus in a 235-node Kubernetes cluster crash-looped because over two million series exhausted host memory and the kernel OOM killer chose it first, as Burstable QoS gave it oom_score_adj 999LINE · 2020

Root cause Capacity exhaustion

IL-0012 CPU limits and an unpatched kernel CFS quota bug throttled multi-threaded services into stalls, failed health checks and timeoutsOmio · 2020

Root cause Software defect

IL-0008 An unresponsive OPA validating webhook crash-looped the API server during a GKE master upgrade, and node auto-repair then recreated every nodeJetstack · 2019

Root cause Configuration change

IL-0002 CI builder pods could not get AWS IAM credentials because kubelet fell minutes behind its API rate limitZalando · 2019

Root cause Capacity exhaustion

IL-0035 Worker nodes went NotReady after a new ElastAlert rule without memory limits used up node memoryNU.nl · 2019

Root cause Configuration change

IL-0059 Single-master Kubernetes control plane kept dying because the kubelet restarted a healthy etcd on a failing kubeadm liveness probeNRE Labs · 2018

Root cause unknown

IL-0074 All AirMap platform services down as worker nodes went NotReady and pods crash-looped on network and DNS failuresAirMap · 2018

Root cause Software defect

IL-0038 kubelet rollout hit a Kubernetes bug and left every node in a cluster NotReady, triggering mass pod evictionOath · 2018

Root cause Software defect

IL-0003 linkerd lost Kubernetes service discovery after an etcd reconfiguration, and restarting every linkerd instance turned failed payments into a full platform outageMonzo · 2017

Root cause Software defect

IL-0040 Parallel Helm installs saturated small gp2 root volumes on a kops cluster and left all three workers NotReady with the kubelet or Docker downSaltside · 2017

Root cause Capacity exhaustion