IL-0031 Zalando · 2018-12-05 (published)kubernetesaws
This is the first of six incidents in Henning Jacobs' talk on running about 100 Kubernetes clusters at Zalando. A team's CronJob manifest, which the talk calls innocent, put concurrencyPolicy: Forbid, successfulJobsHistoryLimit and failedJobsHistoryLimit under the pod template's spec, where they do not belong. The fixed manifest moves those three fields to the CronJob spec, adds activeDeadlineSeconds: 120 to the job spec, and changes the schedule to "7 8-18 * * Mon-Fri". The talk shows this manifest after a slide on the number of pods in the cluster and the API server being OOM-killed.
Because Skipper takes its routes from the API server, the API server outage became an ingress outage: customer traffic failed and the IAM service returned 404. The lessons note that the AWS ALB in front routes traffic to all hosts when all of them report unhealthy. The talk's lessons are to keep the ingress healthy and serving its last known routes during API server problems, and to cap the number of pods with a ResourceQuota.
Customers were affected, and Zalando's IAM service answered requests with HTTP 404. The talk shows this on slides titled "Customer impact" and "IAM returning 404", followed by a slide on the number of pods in the cluster.
The Kubernetes API server had been OOM-killed. Traffic reached applications through an AWS ALB and then Skipper, the HTTP router that Zalando runs as its ingress controller on the nodes, and Skipper loads its routes from the API server. With the API server gone, Skipper lost its source of routes.
The lessons slide adds that when every host behind an AWS ALB reports unhealthy, the ALB sends traffic to all hosts anyway, and the ingress did not stay healthy or keep its last known routes while the API server was down.
Root cause Capacity exhaustion
–
An AWS ALB routes traffic to all hosts when all hosts report unhealthy, so the ingress should stay healthy during API server problems. An ingress controller that reads routes from the API server should keep its last known set of routes. Use a quota for the number of pods.
IL-0170 During a Lunar New Year event, traffic more than three times normal filled the nf_conntrack table on the EKS node running the Istio ingress gateway, dropping packets and taking down Backpackr's web and app servicesBackpackr · 2025same root cause · shared Ingress controller
IL-0204 An etcd memory spike overloaded the control plane of a Render Frankfurt cluster, and DNS then failed because CoreDNS ran on the rebooted control plane nodes and could not start without the API serverRender · 2024same root cause · shared Kubernetes API server
IL-0168 Simultaneous cluster auto-upgrades overloaded Scaleway's Kubernetes API, and a node authentication retry loop kept Kapsule and Kosmos nodes NotReady in FR-PARScaleway · 2024same root cause · shared Kubernetes API server
IL-0195 The kube-apiservers of a cluster with more than 200 nodes ran out of memory and crashed one after another under mass LIST requests from Cilium agentsPalark · 2023same root cause · shared Kubernetes API server
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.
same root cause · shared Kubernetes API server
IL-0194 After a kube-apiserver restart, pod LIST requests from every pod of a Vector DaemonSet nearly tripled etcd memory and restarted all master nodesPalark · 2023same root cause · shared Kubernetes API server
IL-0093 Ingress controller pods crash-looped one after another because the leader pod had exhausted its shared memoryUK Ministry of Justice · 2020same root cause · shared Ingress controller
IL-0002 CI builder pods could not get AWS IAM credentials because kubelet fell minutes behind its API rate limitZalando · 2019same root cause · shared Kubernetes API server
IL-0076 Kubernetes masters went down after Dex filled etcd with AuthRequest objects and deleting its namespace started an endless list-and-timeout loopFullstaq · 2019same root cause · shared Kubernetes API server
IL-0075 Kubernetes API servers pegged on CPU when the cluster grew to about 290 nodes, traced to Datadog agents polling the APICivis Analytics · 2019same root cause · shared Kubernetes API server