IL-0072 Nordstrom · 2017-12-15 (published)kubernetesaws
Nordstrom ran Kubernetes on AWS with a five-node control plane: five etcd instances, five API servers, and schedulers and controller managers in high-availability mode. In etcd, writes are always consistent, but a read is answered from the local member's data unless the client asks for a quorum read. When members fall far behind each other, those local answers are stale, and long replication lag can also cause repeated leader elections. The Kubernetes API server in its default configuration did not request quorum reads, and the high-availability documentation did not mention the setting until October of the year of the talk.
In this incident etcd was hit by high write latency. The API servers returned stale data, so kubectl output, the controller manager and service endpoints flipped between two states. The speakers stress that this was stale data, not a breakdown of etcd's consensus.
The outage on the primary production cluster lasted about four hours, until service was restored on a replacement cluster. Confusion and reluctance to replace the cluster added to the time, because teams used features that are hard to move. Each persistent volume can be attached to only one node, so volumes had to be freed from the old cluster one by one before they could bind in the new one. Some teams had pointed DNS records at the generated names of their Kubernetes load balancers and had to update them by hand to the new load balancers.
The fix is to make the API server request quorum reads. etcd 3.1 makes quorum reads much cheaper because they no longer hit disk, and quorum reads became the default in Kubernetes 1.9. The incident happened during working hours with the full team present, and the root cause analysis led to changes in the team's code and procedures.
A user reported that pods were starting up and then vanishing. When the cluster operators looked, kubectl get pods and kubectl get nodes returned two different sets of data that alternated back and forth, as if there were two opinions of the cluster's state.
The control loops acted on the conflicting data. The controller manager, querying API servers and getting alternating answers, kept trying to converge and spun up and terminated thousands of pods. Service endpoints thrashed, and the ingress controller, responding to the conflicting signals, did very bad things to the traffic that should have flowed through it.
It was a full outage of the primary production cluster: not just down, but violently wrong. The team spent a long time troubleshooting before deciding to replace the cluster.
Symptom Stale or inconsistent API readsStale endpoint routingService outage
A user reported pods starting up and vanishing.
etcd is extremely sensitive to write latency; once the database sync time crosses double digits of milliseconds, stale reads or badly degraded performance follow. Without quorum reads, a consistent store can still hand out stale data.
Documentation for a fast-moving system changes, so read it again regularly. Moving workloads to a new cluster is hard when they hold persistent volumes or when DNS records point at generated load balancer names.
IL-0077 EKS cluster went read-only for over seven hours after Kyverno ephemeralreports filled the etcd database past its size limitMarcin Cuber · 2024 · sev1same root cause · shared Kubernetes API server, etcd
IL-0003 linkerd lost Kubernetes service discovery after an etcd reconfiguration, and restarting every linkerd instance turned failed payments into a full platform outageMonzo · 2017same root cause · shared Kubernetes API server, etcd
IL-0053 Unauthenticated ingress-nginx admission webhook let anyone on the pod network run code and read every Secret in the cluster (IngressNightmare, CVE-2025-1974)Wiz · 2025same root cause · shared Ingress controller
IL-0081 API requests timed out during deployment rollouts because the ingress-nginx enable-serial-reloads option held back endpoint updates while a long NGINX reload was runningCresta · 2025same root cause · shared Ingress controller
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.
same root cause · shared Kubernetes API server
IL-0106 A malformed Traefik IngressRoute triggered a Traefik v2 bug once a node rollover restarted every Traefik pod, and all Kubernetes services returned 404DFDS · 2021same root cause · shared Ingress controller
IL-0190 Cortex distributors kept a stuck connection to a dead etcd leader after a Kubernetes upgrade terminated its node, failing writes to Hosted Prometheus for 12 minutesGrafana Labs · 2020same root cause · shared etcd
IL-0196 Adding a second Redis replica and lowering memory limits in one RedisFailover change restarted the only Redis pod that held data, and Sentinel promoted the empty new replica, erasing the dataPalark · 2020same root cause · shared Persistent volumes and stateful sets
IL-0062 Intermittent 502 errors behind a GKE Ingress because pods shut down before the load balancer stopped sending them trafficRavelin · 2019same root cause · shared Ingress controller
IL-0011 HAProxy ingress piled requests onto the same pods because a Go template visited the backend map in sorted pod IP orderSkyscanner · 2019same root cause · shared Ingress controller
IL-0030 Most nodes of a cluster went NotReady together for 15 minutes while the API server sat behind an AWS ELBZalando · 2019same root cause · shared Kubernetes API server