IL-0038 Oath · 2018-05-04 (published)kubernetes
Oath ran Kubernetes on premises for media sites such as its sports, finance and home pages, in 12 production clusters across six regions behind anycast routing. While rolling out a kubelet change in one cluster, kubelet hit a Kubernetes bug during its cgroup cleanup at restart and did not come back. The rolling update kept going batch by batch with no check that the updated kubelets worked, so it reached the whole cluster.
With every kubelet down, the API server saw every node as NotReady and the node controller began evicting pods. Its built-in protection limited the damage: above 50% NotReady nodes it slowed eviction, and with all nodes down it entered full disruption mode and stopped evicting altogether. About half the pods had been evicted by then; the rest kept serving because the pods and the ingress path were healthy. The team failed the data center over so users were served from the next one.
During a rolling update of a kubelet change in one of Oath's on-premises production clusters, nodes started going NotReady, and soon all nodes in the cluster were NotReady.
The graphs showed a growing number of nodes entering the eviction queue and pods being evicted from them. The node controller in the Kubernetes controller manager tries to move pods off a NotReady node so they can run elsewhere, but with no Ready nodes left the evicted pods could only sit in Pending.
The workloads themselves were still running: only the link between kubelet and the API server was broken, and traffic from the ingress to the pods still worked. Once half of the nodes were NotReady, the node controller slowed its eviction rate, and when it saw the whole cluster down it entered full disruption mode and stopped evicting. By then about half of the pods had been evicted, so users saw a degraded service rather than a full outage.
Root cause Software defect
Nodes reported NotReady and the eviction graphs climbed while the kubelet rollout was in progress.
Check kubelet after upgrading it before moving to the next batch; a simple curl against the kubelet API would have caught this. The node controller's eviction throttling and full disruption mode, which the speakers note is not well documented, kept a cluster-wide kubelet failure from evicting every pod.
IL-0112 The daily node rollover put DFDS's EKS nodes on an AMI later recalled for a kernel issue, causing "PLEG is not healthy" NodeNotReady flaps and pod network timeoutsDFDS · 2022same root cause · shared kubelet
IL-0139 Pods on GKE clusters with Calico network policy got stuck Terminating or Pending after upgrading to 1.22 or later because of a race condition in the Calico componentGoogle Cloud · 2022same root cause · shared kubelet
IL-0012 CPU limits and an unpatched kernel CFS quota bug throttled multi-threaded services into stalls, failed health checks and timeoutsOmio · 2020same root cause · shared kubelet
IL-0074 All AirMap platform services down as worker nodes went NotReady and pods crash-looped on network and DNS failuresAirMap · 2018same root cause · shared kubelet
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.
same root cause · shared kubelet