IL-0030 Zalando · 2019-02-11 (published)kubernetesaws
Henning Jacobs of Zalando presented this as the first of five failure stories at a meetup in Hamburg on 2019-02-11. At the time Zalando counted 100 clusters and 373 accounts. This case covers only that first story; the CoreDNS outage from the same talk is a separate case.
On a cluster running Kubernetes v1.7.4 (CoreOS build), most nodes turned NotReady together, leaving fewer than 20% of nodes available. All of them recovered on their own after 15 minutes. The clues the team listed were that failure and recovery happened at the same time on every node, that the API server sat behind an AWS ELB, and that other users reported the same symptom, some of them about monthly.
The team matched it to the upstream issue kubernetes/kubernetes#48638, which was fixed in Kubernetes 1.8 and backported to 1.7.8.
Fewer than 20% of the nodes in one Zalando cluster were available. Most nodes reported NotReady at the same moment, and only a handful stayed Ready.
The kubectl get nodes output shown in the talk lists nodes such as ip-172-31-10-91...internal as NotReady, age 4d, version v1.7.4+coreos.0. In that excerpt five of eleven nodes are NotReady, four are Ready and two are Ready,SchedulingDisabled.
The nodes did not fail one by one. They all became unhealthy at the same time and all recovered at the same time, about 15 minutes later, without anyone intervening. Other Kubernetes users reported the same pattern, and some of them saw it roughly once a month.
Symptom Node NotReady
Root cause Software defect
–
–
–
IL-0150 A latent race condition in DynamoDB's DNS automation left an empty DNS record for dynamodb.us-east-1.amazonaws.com, and the DynamoDB and EC2 launch failures that followed caused EKS, ECS and Fargate container launch failures and cluster scaling delays in us-east-1Amazon Web Services · 2025same root cause · shared Cloud load balancer
IL-0138 A failed Nvidia driver build left broken kernel packages on the only Kubernetes master node, which hit a kernel panic on reboot after a power outage and took the homelab cluster down for six daysBarush Mendez · 2025same root cause · shared Kubernetes API server
IL-0131 A new kind of GenerateContent request made Vertex AI prediction API servers segfault, and once enough of them restarted at the same time the serving fleet could not carry the loadGoogle Cloud · 2024same root cause · shared Cloud load balancer
IL-0077 EKS cluster went read-only for over seven hours after Kyverno ephemeralreports filled the etcd database past its size limitMarcin Cuber · 2024 · sev1same root cause · shared Kubernetes API server
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.
same root cause · shared Cloud load balancer
IL-0072 High etcd write latency made the API servers return stale data, and the production cluster flipped between two views of its stateNordstrom · 2017same root cause · shared Kubernetes API server
IL-0003 linkerd lost Kubernetes service discovery after an etcd reconfiguration, and restarting every linkerd instance turned failed payments into a full platform outageMonzo · 2017same root cause · shared Kubernetes API server