IL-0195 Palark · 2023-06-07 (published)kubernetes
A customer cluster run by Palark, with more than 200 nodes, lost its Kubernetes API when the Cilium agent pods sent LIST requests to the API en masse. Simultaneous LIST requests from so many nodes drove kube-apiserver memory up until one API server used all the memory on its control plane node and crashed; its requests then moved to the next API server, which failed the same way.
To keep production running, the team first added CPU and memory to the control plane nodes. The first increase was not enough, and the API stabilised only after the next one; in total the nodes went from 8 CPUs and 16 GB of RAM to 16 CPUs and 64 GB.
The lasting fix used API Priority and Fairness (APF), the API server feature that sorts incoming requests into priority levels, each with its own share of concurrency and its own queues. A FlowSchema named cilium-pods matched LIST requests to the cilium.io API group from the group system:serviceaccounts:d8-cni-cilium, split them into flows by user, and sent them to a PriorityLevelConfiguration of type Limited with assuredConcurrencyShares 5, 16 queues, handSize 4 and queueLengthLimit 50. The team accepted slightly slower LIST responses for Cilium because they would not affect Cilium's work. With the new objects in place, restarting cilium-agent no longer changed memory use significantly, and the control plane nodes went back to their original size.
A tech support engineer called Palark to say that a customer's cluster, production environment included, had practically stopped working. On the control plane nodes the kube-apiserver processes had used up all the memory and were crashing and restarting over and over, so the Kubernetes API was unreachable and completely non-functional.
Memory graphs from the time of the failure showed memory consumption as high as 50 GB. The control plane nodes originally had 8 CPUs and 16 GB of RAM each. In a reply to a reader, the team explains how the source was found: a Grafana dashboard of API request counts showed a large increase, the Prometheus query apiserver_request_total{verb=~"LIST", instance=~".*:.*", resource=~".*"} grouped by instance and resource showed that many LIST requests came from Cilium network policies, and the API audit log showed the individual requests.
In a test window agreed with the customer, the team restarted the Cilium agents and watched the same sequence repeat: the load on one API server grew, it consumed all the memory on its node and crashed, the requests moved to another API server, and the same thing happened there.
Root cause Capacity exhaustion
A tech support engineer phoned the team to report that the customer's cluster was practically non-operational.
In a large cluster, many agents sending LIST requests at once can exhaust kube-apiserver memory, and with several API servers the crash moves from one to the next. APF can hold back such a client: a FlowSchema matches its requests and a PriorityLevelConfiguration gives them a small concurrency share and a queue, so the requests run more slowly instead of all at once.
With APF enabled, the total concurrency it divides is the sum of --max-requests-inflight (default 400) and --max-mutating-requests-inflight (default 200), shared among priority levels in proportion to their assured concurrency shares, so raising one level's share lowers every other level's limit. Long-running requests such as logs and exec, and WATCH requests, are not subject to APF. The response headers X-Kubernetes-PF-FlowSchema-UID and X-Kubernetes-PF-PriorityLevel-UID show where a request landed, and the metrics Apiserver_flowcontrol_rejected_requests_total, Apiserver_current_inqueue_requests and Apiserver_flowcontrol_request_execution_seconds, with the debug endpoints /debug/api_priority_and_fairness/dump_priority_levels and /debug/api_priority_and_fairness/dump_queues, show how the queues behave.
IL-0204 An etcd memory spike overloaded the control plane of a Render Frankfurt cluster, and DNS then failed because CoreDNS ran on the rebooted control plane nodes and could not start without the API serverRender · 2024same root cause · shared Kubernetes API server
IL-0168 Simultaneous cluster auto-upgrades overloaded Scaleway's Kubernetes API, and a node authentication retry loop kept Kapsule and Kosmos nodes NotReady in FR-PARScaleway · 2024same root cause · shared Kubernetes API server
IL-0165 Some LKE users in Frankfurt could not reach their kube-apiservers with kubectl after backend services failed under memory constraintsLinode · 2023same root cause · shared Kubernetes API server
IL-0194 After a kube-apiserver restart, pod LIST requests from every pod of a Vector DaemonSet nearly tripled etcd memory and restarted all master nodesPalark · 2023same root cause · shared Kubernetes API server
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.
same root cause · shared Kubernetes API server
IL-0076 Kubernetes masters went down after Dex filled etcd with AuthRequest objects and deleting its namespace started an endless list-and-timeout loopFullstaq · 2019same root cause · shared Kubernetes API server
IL-0075 Kubernetes API servers pegged on CPU when the cluster grew to about 290 nodes, traced to Datadog agents polling the APICivis Analytics · 2019same root cause · shared Kubernetes API server
IL-0031 Kubernetes API server OOM-killed, leaving the Skipper ingress without routes; the talk points to a misconfigured CronJob manifestZalando · 2018same root cause · shared Kubernetes API server