Cases5 cases

1 to 5 of 5

1. IL-0204 An etcd memory spike overloaded the control plane of a Render Frankfurt cluster, and DNS then failed because CoreDNS ran on the rebooted control plane nodes and could not start without the API server

   Render2024

   Symptoms

   In January 2022 an etcd memory spike overloaded the Kubernetes control plane of one of Render's clusters in Frankfurt. At first only operations that need the control plane failed: new deploys and cron job runs.

   Control plane unresponsiveDeploys failingDNS resolution failuresHigh error rate

   Trigger

   A memory spike in etcd overloaded the cluster's control plane.

   Root cause

   Capacity exhaustion

   Component

   etcdCoreDNSKubernetes API server

   Contributing factor

   Unreviewed defaultsHarmful manual action

   Action items

   • Run CoreDNS on data plane nodes instead of control plane nodes
   • Run etcd on dedicated nodes

   Lessons

   CoreDNS can keep answering from its in-memory cache while the API server is down, but it cannot initialise without it. With CoreDNS on the control plane nodes, a reboot or restart during a control plane outage turned a failure of deploys and cron jobs into a failure of requests to services that were already running. The core problem was that the data plane was not isolated enough from the control plane: even without the reboot, the etcd memory spike alone could have forced CoreDNS to restart.

   View case
2. IL-0195 The kube-apiservers of a cluster with more than 200 nodes ran out of memory and crashed one after another under mass LIST requests from Cilium agents

   Palark2023

   Symptoms

   A tech support engineer called Palark to say that a customer's cluster, production environment included, had practically stopped working. On the control plane nodes the kube-apiserver processes had used up all the memory and were crashing and restarting over and over, so the Kubernetes API was unreachable and completely non-functional.

   Control plane unresponsiveOOMKilled

   Trigger

   Cilium agent pods on more than 200 nodes sent LIST requests to the Kubernetes API en masse at the same time.

   Root cause

   Capacity exhaustion

   Component

   Kubernetes API serverCilium

   Contributing factor

   Unreviewed defaults

   Detection

   A tech support engineer phoned the team to report that the customer's cluster was practically non-operational.

   Remediation

   • added CPU and memory to the control plane nodes in two steps, from 8 CPUs and 16 GB of RAM to 16 CPUs and 64 GB, until the API stabilised
   • restarted the Cilium agents in a test window agreed with the customer to reproduce the memory growth
   • deployed a FlowSchema and a PriorityLevelConfiguration named cilium-pods that limit and queue LIST requests from the Cilium service accounts to cilium.io resources
   • reduced the control plane nodes back to their original resources

   Action items

   • Make an API Priority and Fairness configuration a standard part of Palark's Kubernetes platform

   Lessons

   In a large cluster, many agents sending LIST requests at once can exhaust kube-apiserver memory, and with several API servers the crash moves from one to the next. APF can hold back such a client: a FlowSchema matches its requests and a PriorityLevelConfiguration gives them a small concurrency share and a queue, so the requests run more slowly instead of all at once. With APF enabled, the total concurrency it divides is the sum of --max-requests-inflight (default 400) and --max-mutating-requests-inflight (default 200), shared among priority levels in proportion to their assured concurrency shares, so raising one level's share lowers every other level's limit. Long-running requests such as logs and exec, and WATCH requests, are not subject to APF. The response headers X-Kubernetes-PF-FlowSchema-UID and X-Kubernetes-PF-PriorityLevel-UID show where a request landed, and the metrics Apiserver_flowcontrol_rejected_requests_total, Apiserver_current_inqueue_requests and Apiserver_flowcontrol_request_execution_seconds, with the debug endpoints /debug/api_priority_and_fairness/dump_priority_levels and /debug/api_priority_and_fairness/dump_queues, show how the queues behave.

   View case
3. IL-0194 After a kube-apiserver restart, pod LIST requests from every pod of a Vector DaemonSet nearly tripled etcd memory and restarted all master nodes

   Palark2023

   Symptoms

   One evening, right after an ordinary restart of kube-apiserver in a customer's production cluster, etcd memory consumption rose almost threefold. The sudden growth led to a cascading restart of all the master nodes.

   Control plane unresponsiveDatabase overload

   Trigger

   An ordinary restart of kube-apiserver, after which every pod of the log collector DaemonSet sent its list requests at the same time.

   Root cause

   Capacity exhaustion

   Component

   etcdKubernetes API serverTelemetry agent

   Contributing factor

   Unknown limitUnreviewed defaults

   Detection

   Memory graphs showed etcd memory on the master nodes almost tripling after a kube-apiserver restart; the audit logs then pointed to one DaemonSet's list requests.

   Remediation

   • created a PriorityLevelConfiguration named limit-list-custom that queues requests (assuredConcurrencyShares 5, 16 queues, handSize 4, queueLengthLimit 50)
   • created a FlowSchema that routes list and get requests on pods from the log collector's service account to that priority level

   Action items

   • Submit a pull request adding resource version support to the Kubernetes Rust library
   • Open a pull request to the Vector log collector to address the startup list requests
   • Apply the same flow-control approach to a similar problem found with the Cilium CNI

   Lessons

   A field selector reduces what kube-apiserver returns, not what it reads from etcd: listing the pods of one node still reads every pod in the cluster. A DaemonSet whose pods all list at startup multiplies that by the node count each time kube-apiserver restarts. Where exact freshness is not needed, resourceVersion=0 lets kube-apiserver answer from its cache; where the client cannot set it, API Priority and Fairness can queue the requests so they reach etcd gradually.

   View case
4. IL-0002 CI builder pods could not get AWS IAM credentials because kubelet fell minutes behind its API rate limit

   Zalando2019

   Symptoms

   Builds on Zalando's in-house Continuous Delivery Platform (CDP), which runs CI/CD on Kubernetes, timed out and failed for almost an hour on 11 June 2019. The Kubernetes builder pods could not get AWS IAM credentials.

   High error rateCredentials unavailableControl plane lag

   Trigger

   An unlucky rescaling of the CDP Kubernetes cluster left only one node available for builder pods. Builder pods were created and deleted on that node so quickly that kubelet fell behind.

   Root cause

   Capacity exhaustion

   Component

   kubeletCredential broker for podsKubernetes API serverCI system

   Contributing factor

   Unreviewed defaultsSingle point of failure

   Remediation

   • Scaled the CDP cluster up by hand so builder pods spread over more than one node

   Action items

   • Raise the kubelet QPS limit from the default 5 to 50 in the global cluster configuration

   View case
5. IL-0076 Kubernetes masters went down after Dex filled etcd with AuthRequest objects and deleting its namespace started an endless list-and-timeout loop

   Fullstaq2019

   Symptoms

   Dex, the OIDC proxy that handled Kubernetes authentication, stored its data in the Kubernetes API (storage type kubernetes, inCluster: true) and had built up a huge number of AuthRequest objects. Listing all AuthRequests timed out, and without the list they could not be deleted.

   Control plane unresponsiveMonitoring outage

   Trigger

   The dex namespace was deleted so that Kubernetes would clean up the accumulated AuthRequest objects itself.

   Root cause

   Capacity exhaustion

   Component

   Kubernetes API serveretcdCluster authentication provider

   Contributing factor

   Unreviewed defaultsHarmful manual actionRetry stormLimited debug accessIgnored early signal

   Detection

   The speaker says he had got hints of the problem earlier. While the masters were down, plenty of alerts arrived in Slack, but monitoring could not be reached because authentication and port-forwarding no longer worked.

   Remediation

   • removed the kubernetes finalizer from the dex namespace by calling the API directly (kill-kube-ns)
   • deployed Dex to a new namespace with sane defaults

   Lessons

   Dex with Kubernetes as its storage keeps every AuthRequest as an API object, and with a long TTL they pile up until the API cannot list them. Deleting a namespace that holds that many objects makes Kubernetes list them in a loop and can overload the masters. Monitoring that is reached through cluster authentication is lost when authentication is what broke.

   View case

By root cause1