IL-0194 Palark · 2023-03-23kubernetes
The DaemonSet was the Vector log collector, which uses pod data to add metadata to log lines. Each of its pods asked kube-apiserver for the pods on its own node with a field selector, a request that returns at most 110 pods. etcd, however, stores objects as plain key-value pairs under /registry/<kind>/<namespace>/<name> and knows nothing about field or label selectors, so kube-apiserver had to read every pod in the cluster from etcd (7130 in the logged request) and filter them itself. With the default resourceVersion behaviour (MostRecent), every such list goes to etcd. When all the collector pods started together after a kube-apiserver restart, these reads drove etcd memory up and brought the master nodes down.
Palark first tried resourceVersion=0, which lets kube-apiserver answer from its cache: a test list of all pods took 117 ms instead of 337 ms. The collector is written in Rust, and the Kubernetes Rust library had no resource version setting, so this was not available right away; Palark later submitted a pull request to the library. To end the incident, Palark used API Priority and Fairness to queue the requests instead. A PriorityLevelConfiguration named limit-list-custom (type Limited, assuredConcurrencyShares 5, queuing with 16 queues, handSize 4, queueLengthLimit 50) and a FlowSchema that sends list and get requests on pods from the collector's service account to that level, distinguished by user, were applied. After that, etcd memory was mostly normal with some spikes left, and the incident's criticality was lowered.
Palark found a similar problem with the Cilium CNI and handled it the same way.
One evening, right after an ordinary restart of kube-apiserver in a customer's production cluster, etcd memory consumption rose almost threefold. The sudden growth led to a cascading restart of all the master nodes.
etcd logged slow reads over the whole pod key range, for example a warning "apply request took too long" with "took": "130.768157ms", "expected-duration": "100ms", a read-only range from "/registry/pods/" to "/registry/pods0" and "range_response_count:7130 size:13313287".
The audit logs pointed to one DaemonSet. Whenever kube-apiserver restarted, each of its pods sent new list requests to fill its caches, which is the default behaviour of client-go informers: 60 list requests per pod at startup, with about 80 nodes in the cluster. The requests had the form /api/v1/pods?fieldSelector=spec.nodeName=$NODE_NAME. Redeploying the pods one after another left no mark on the memory graphs; only restarting them all together did.
Root cause Capacity exhaustion
Memory graphs showed etcd memory on the master nodes almost tripling after a kube-apiserver restart; the audit logs then pointed to one DaemonSet's list requests.
A field selector reduces what kube-apiserver returns, not what it reads from etcd: listing the pods of one node still reads every pod in the cluster. A DaemonSet whose pods all list at startup multiplies that by the node count each time kube-apiserver restarts. Where exact freshness is not needed, resourceVersion=0 lets kube-apiserver answer from its cache; where the client cannot set it, API Priority and Fairness can queue the requests so they reach etcd gradually.
IL-0075 Kubernetes API servers pegged on CPU when the cluster grew to about 290 nodes, traced to Datadog agents polling the APICivis Analytics · 2019same root cause · shared Kubernetes API server, Telemetry agent, etcd
IL-0204 An etcd memory spike overloaded the control plane of a Render Frankfurt cluster, and DNS then failed because CoreDNS ran on the rebooted control plane nodes and could not start without the API serverRender · 2024same root cause · shared Kubernetes API server, etcd
IL-0076 Kubernetes masters went down after Dex filled etcd with AuthRequest objects and deleting its namespace started an endless list-and-timeout loopFullstaq · 2019same root cause · shared Kubernetes API server, etcd
IL-0042 Application logs stopped reaching Elasticsearch and OpenSearch for hours and were lost when OpenSearch ran out of disk spaceUK Ministry of Justice · 2024same root cause · shared Telemetry agent
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.
same root cause · shared Telemetry agent
IL-0168 Simultaneous cluster auto-upgrades overloaded Scaleway's Kubernetes API, and a node authentication retry loop kept Kapsule and Kosmos nodes NotReady in FR-PARScaleway · 2024same root cause · shared Kubernetes API server
IL-0043 Application logs missing in Kibana after a newer Fluent Bit version changed its chunk drop strategyUK Ministry of Justice · 2023same root cause · shared Telemetry agent
IL-0195 The kube-apiservers of a cluster with more than 200 nodes ran out of memory and crashed one after another under mass LIST requests from Cilium agentsPalark · 2023same root cause · shared Kubernetes API server
IL-0165 Some LKE users in Frankfurt could not reach their kube-apiservers with kubectl after backend services failed under memory constraintsLinode · 2023same root cause · shared Kubernetes API server
IL-0026 Kubernetes nodes hit SystemOOM because fluentd-sumologic log shipper pods had no memory limitsBlue Matador · 2019same root cause · shared Telemetry agent
IL-0002 CI builder pods could not get AWS IAM credentials because kubelet fell minutes behind its API rate limitZalando · 2019same root cause · shared Kubernetes API server
IL-0031 Kubernetes API server OOM-killed, leaving the Skipper ingress without routes; the talk points to a misconfigured CronJob manifestZalando · 2018same root cause · shared Kubernetes API server