Cases3 cases

1 to 3 of 3

1. IL-0168 Simultaneous cluster auto-upgrades overloaded Scaleway's Kubernetes API, and a node authentication retry loop kept Kapsule and Kosmos nodes NotReady in FR-PAR

   Scaleway2024

   Symptoms

   At 01:04 CET on 13 February 2024, the Kubernetes API behind Scaleway's managed Kubernetes services Kapsule and Kosmos in the Paris region (FR-PAR) spiked in RAM and CPU use and ran out of memory. Nodes in FR-PAR could no longer authenticate to their control plane, and some of them went "Not Ready". Customers on both shared (mutualized) and dedicated environments were affected.

   Node NotReadyControl plane lagOOMKilled

   Trigger

   A new Kubernetes release was made available to all users on 12 February 2024, and during the night of 12 to 13 February many clusters triggered their auto-upgrade at the same time.

   Root cause

   Capacity exhaustion

   Component

   Kubernetes API server

   Contributing factor

   Retry stormAutomation amplificationNo staged rollout

   Detection

   The timeline's first entry, at 01:04 CET, records the Kubernetes API's RAM and CPU spikes and out-of-memory state, with nodes unable to authenticate to their control plane and some becoming "Not Ready". The first recorded action, 42 minutes later at 01:46, was to increase the resources allocated to the API.

   Remediation

   • increased the resources allocated to the Kubernetes API at 01:46 and again at 02:20
   • applied API traffic restrictions at 02:30 to smooth the incoming traffic
   • removed the traffic restrictions progressively in 50-100 batches from 03:20
   • ran a script at 03:40 to replace the failing nodes created during the congestion
   • rolled back to the former node authentication mechanism, first on a subset of clusters from 15:45 and on all clusters by 19:41 on 19 February
   • disabled cluster upgrade and configuration update features until 10:30 on 20 February

   Action items

   • Increase the resources allocated to the Kubernetes API
   • Roll back to the former node authentication mechanism
   • Implement a local authentication cache on nodes
   • Rework the resource allocation strategy to behave better under load spikes
   • Add new request queuing mechanisms for Kubernetes
   • Separate the different components to reduce side effects under high load or failure
   • Monitor selected metrics closely to identify anomalous patterns earlier
   • Refund customers who deployed nodes during the incident but could not use them normally

   Lessons

   Letting many clusters auto-upgrade at the same moment after a release turns routine upgrades into one coordinated load spike on the shared Kubernetes API. A node authentication mechanism that retries without letting up turns a slow API into an unavailable one, and auto-healing that replaces NotReady nodes amplifies the problem. The congestion pattern had already appeared three weeks earlier, and short-term mitigations did not stop it from recurring, so Scaleway rolled back the new authentication mechanism. Scaleway notes that as the number of customer clusters and nodes grows sharply, traffic control mechanisms have to improve to prevent congestion.

   View case
2. IL-0195 The kube-apiservers of a cluster with more than 200 nodes ran out of memory and crashed one after another under mass LIST requests from Cilium agents

   Palark2023

   Symptoms

   A tech support engineer called Palark to say that a customer's cluster, production environment included, had practically stopped working. On the control plane nodes the kube-apiserver processes had used up all the memory and were crashing and restarting over and over, so the Kubernetes API was unreachable and completely non-functional.

   Control plane unresponsiveOOMKilled

   Trigger

   Cilium agent pods on more than 200 nodes sent LIST requests to the Kubernetes API en masse at the same time.

   Root cause

   Capacity exhaustion

   Component

   Kubernetes API serverCilium

   Contributing factor

   Unreviewed defaults

   Detection

   A tech support engineer phoned the team to report that the customer's cluster was practically non-operational.

   Remediation

   • added CPU and memory to the control plane nodes in two steps, from 8 CPUs and 16 GB of RAM to 16 CPUs and 64 GB, until the API stabilised
   • restarted the Cilium agents in a test window agreed with the customer to reproduce the memory growth
   • deployed a FlowSchema and a PriorityLevelConfiguration named cilium-pods that limit and queue LIST requests from the Cilium service accounts to cilium.io resources
   • reduced the control plane nodes back to their original resources

   Action items

   • Make an API Priority and Fairness configuration a standard part of Palark's Kubernetes platform

   Lessons

   In a large cluster, many agents sending LIST requests at once can exhaust kube-apiserver memory, and with several API servers the crash moves from one to the next. APF can hold back such a client: a FlowSchema matches its requests and a PriorityLevelConfiguration gives them a small concurrency share and a queue, so the requests run more slowly instead of all at once. With APF enabled, the total concurrency it divides is the sum of --max-requests-inflight (default 400) and --max-mutating-requests-inflight (default 200), shared among priority levels in proportion to their assured concurrency shares, so raising one level's share lowers every other level's limit. Long-running requests such as logs and exec, and WATCH requests, are not subject to APF. The response headers X-Kubernetes-PF-FlowSchema-UID and X-Kubernetes-PF-PriorityLevel-UID show where a request landed, and the metrics Apiserver_flowcontrol_rejected_requests_total, Apiserver_current_inqueue_requests and Apiserver_flowcontrol_request_execution_seconds, with the debug endpoints /debug/api_priority_and_fairness/dump_priority_levels and /debug/api_priority_and_fairness/dump_queues, show how the queues behave.

   View case
3. IL-0031 Kubernetes API server OOM-killed, leaving the Skipper ingress without routes; the talk points to a misconfigured CronJob manifest

   Zalando2018

   Symptoms

   Customers were affected, and Zalando's IAM service answered requests with HTTP 404. The talk shows this on slides titled "Customer impact" and "IAM returning 404", followed by a slide on the number of pods in the cluster.

   High error rateControl plane unresponsiveOOMKilled

   Trigger

   A CronJob (apiVersion batch/v2alpha1, schedule "*/15 9-19 * * Mon-Fri") whose concurrencyPolicy: Forbid and job history limits sat inside the pod template spec instead of the CronJob spec. The talk shows it right after the slides on the number of pods and the API server being OOM-killed, and the lessons add a quota on the number of pods.

   Root cause

   Capacity exhaustion

   Component

   Kubernetes API serverIngress controller

   Contributing factor

   Single point of failureExcessive blast radius

   Remediation

   • fixed the CronJob manifest: moved concurrencyPolicy: Forbid and the job history limits to the CronJob spec, added activeDeadlineSeconds: 120 and changed the schedule to "7 8-18 * * Mon-Fri"

   Action items

   • Make the ingress stay healthy during API server problems
   • Make the ingress keep the last known set of routes
   • Use a ResourceQuota on the number of pods (the example sets pods: "1500")

   Lessons

   An AWS ALB routes traffic to all hosts when all hosts report unhealthy, so the ingress should stay healthy during API server problems. An ingress controller that reads routes from the API server should keep its last known set of routes. Use a quota for the number of pods.

   View case

By root cause1