Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Symptom

Node NotReady

노드 NotReady · node-not-ready

23 cases · 2017 to 2026

Root cause Capacity exhaustion9 Software defect5 Cloud provider outage2 Configuration change2 Dependency failure2 DNS or network fault1 Hardware failure1 Breaking change in an upgrade1

Component kubelet8 Node operating system image5 Application database4 Linux kernel4

Contributing factor Automation amplification7 Excessive blast radius6 Single point of failure6 Unverified assumption5

Organisation DigitalOcean3 DFDS2 Scaleway2 AirMap1 Buffer1

Cases

IL-0155 Some DOKS clusters in NYC1 saw intermittent DNS timeouts and NodeNotReady events because CoreDNS ran on shared-CPU Droplet worker nodesDigitalOcean · 2026
Root cause Capacity exhaustion
IL-0085 A control plane node of Magic Pages' Kubernetes cluster on Hetzner Cloud ran out of memory, the cluster's own calls then used up the Hetzner API rate limit, and all customer websites were down for about 7.5 hoursMagic Pages · 2025
Root cause Capacity exhaustion
Line cards on an SFO3 core router rebooted together and halved backbone capacity, and DOKS nodes whose guest networking did not recover stayed NotReady with degraded API servers until all affected nodes in the SFO3 DOKS fleet were recycled

View on the graph

Container memory limit4

Kubernetes API server3

Node root disk3

Container runtime2

Persistent volumes and stateful sets2

Pod scheduling and placement2

Telemetry agent2

Cloud controller manager1

Cloud load balancer1

ConfigMaps and Secrets1

Container CPU limit1

Credential broker for pods1

Jobs and CronJobs1

Kubernetes controller manager1

Node launch template1

Service discovery1

systemd-networkd1

Unpatched known bug5

Monitoring blind spot5

Unreviewed defaults4

No staged rollout3

Change with a delayed effect2

Limited debug access2

No alert on limit2

No health probes2

No resource requests or limits2

Overlapping failures2

Ignored early signal1

Harmful manual action1

UK Ministry of Justice1

IL-0154

DigitalOcean · 2024

Root cause DNS or network fault

IL-0110 A Grafana Agent pod's high memory use and crash-looping Hangfire pods drove a Kubernetes node into system OOM and NotReady, twice in one dayDFDS · 2024

Root cause Capacity exhaustion

IL-0021 GKE GPU nodes on Container-Optimized OS could not download NVIDIA drivers (403 Forbidden) and came up without working GPUsGoogle Cloud · 2024

Root cause Dependency failure

IL-0168 Simultaneous cluster auto-upgrades overloaded Scaleway's Kubernetes API, and a node authentication retry loop kept Kapsule and Kosmos nodes NotReady in FR-PARScaleway · 2024

Root cause Capacity exhaustion

IL-0169 The Scaleway Kubernetes API in FR-PAR ran out of memory under simultaneous deployments, and because it also handled node authentication, Kapsule and Kosmos nodes went NotReadyScaleway · 2024

Root cause Capacity exhaustion

IL-0153 DOKS worker nodes went NotReady at 06:00 UTC on three days because a systemd timer in the Debian image ran apt upgrade and killed it mid kernel upgrade, breaking node networkingDigitalOcean · 2023

Root cause Configuration change

IL-0016 Prometheus on the live cluster ran out of memory and was OOMKilled, and the monitoring node it ran on went NotReadyUK Ministry of Justice · 2023

Root cause Capacity exhaustion

IL-0185 A disk failure made the first control plane node of a Kubespray cluster read-only, and recovery needed the Kubespray inventory reordered and a new control plane node joinedHans H. Kim · 2023

Root cause Hardware failure

IL-0198 An automatic Ubuntu security update restarted systemd-networkd, which flushed the IP rules Cilium relied on and cut more than 60 percent of Datadog's Kubernetes nodes off the network across regions and cloud providersDatadog · 2023

Root cause Breaking change in an upgrade

IL-0112 The daily node rollover put DFDS's EKS nodes on an AMI later recalled for a kernel issue, causing "PLEG is not healthy" NodeNotReady flaps and pod network timeoutsDFDS · 2022

Root cause Software defect

IL-0172 An AWS Tokyo data centre cooling failure took down 6 of 60 CockroachDB nodes on Kubernetes within 22 minutes, and ranges lost quorum because the Helm-deployed cluster had no availability zone locality configuredDevsisters · 2021

Root cause Cloud provider outage

IL-0029 Slow user-facing services from CPU throttling on containers that used far less CPU than their limitBuffer · 2020

Root cause Software defect

IL-0193 A GCP persistent disk degradation took down a Cassandra StatefulSet whose OrderedReady policy blocked recovery, and Metrictank could not reconnect to the recreated pods, causing a 23-hour Graphite query outageGrafana Labs · 2019

Root cause Cloud provider outage

IL-0030 Most nodes of a cluster went NotReady together for 15 minutes while the API server sat behind an AWS ELBZalando · 2019

Root cause Software defect

IL-0035 Worker nodes went NotReady after a new ElastAlert rule without memory limits used up node memoryNU.nl · 2019

Root cause Configuration change

IL-0064 CPU-heavy pods packed onto one Google Cloud node drove it to kernel panics, and the crash moved from node to node until the site was downMoonlight · 2019

Root cause Capacity exhaustion

IL-0073 Intermittent Kafka connectivity overloaded Docker daemons, set off a node rescheduling cascade and poisoned the Consul gossip meshTarget · 2019 · sev1

Root cause Dependency failure

IL-0074 All AirMap platform services down as worker nodes went NotReady and pods crash-looped on network and DNS failuresAirMap · 2018

Root cause Software defect

IL-0038 kubelet rollout hit a Kubernetes bug and left every node in a cluster NotReady, triggering mass pod evictionOath · 2018

Root cause Software defect

IL-0036 A reindex Job that kept failing exhausted one node after another until the whole cluster was downUniverse · 2017

Root cause Capacity exhaustion

IL-0040 Parallel Helm installs saturated small gp2 root volumes on a kops cluster and left all three workers NotReady with the kubelet or Docker downSaltside · 2017

Root cause Capacity exhaustion