Cases28 cases

11 to 15 of 28

1. IL-0152 An incorrect configuration applied during maintenance made an internal infrastructure cluster automatically delete the control planes of 200 DOKS clusters in FRA1

   DigitalOcean2024

   Symptoms

   At 13:02 UTC on 2024-01-23 DigitalOcean posted that, as of 12:18 UTC, its engineers were investigating an issue with Kubernetes clusters in the FRA1 region and that users might see errors when communicating with their clusters. The control planes of the affected DOKS (DigitalOcean Kubernetes) clusters had become inaccessible.

   Resources deletedControl plane unresponsive

   Trigger

   An incorrect configuration was applied to an internal infrastructure cluster in FRA1 at 12:21 UTC during a maintenance operation.

   Root cause

   Configuration change

   Component

   etcdKubernetes API server

   Contributing factor

   Automation amplificationNo safeguard on destructive changeExcessive blast radius

   Detection

   Engineers were investigating errors with Kubernetes clusters in FRA1 as of 12:18 UTC and identified the incorrect configuration as the cause at 12:36.

   Remediation

   • restored each affected cluster's etcd from its latest twice-daily backup
   • restored the stateless control plane components after etcd
   • patched the recovery mechanism after unexpected problems during recovery
   • found and fixed an additional small bug during recovery
   • contacted affected customers directly with further instructions, since some might need to reconfigure Kubernetes resources

   Action items

   • Prevent automatic deletion of infrastructure cluster resources under any circumstances in the production environment

   Lessons

   When managed control planes run as resources in another cluster, one configuration mistake in that cluster can delete hundreds of control planes at once through ordinary automation. The etcd backups limited the damage, but their twice-daily schedule set the worst case at 12 hours of lost control plane state, and the recovery tooling itself needed patching during the incident.

   View case
2. IL-0195 The kube-apiservers of a cluster with more than 200 nodes ran out of memory and crashed one after another under mass LIST requests from Cilium agents

   Palark2023

   Symptoms

   A tech support engineer called Palark to say that a customer's cluster, production environment included, had practically stopped working. On the control plane nodes the kube-apiserver processes had used up all the memory and were crashing and restarting over and over, so the Kubernetes API was unreachable and completely non-functional.

   Control plane unresponsiveOOMKilled

   Trigger

   Cilium agent pods on more than 200 nodes sent LIST requests to the Kubernetes API en masse at the same time.

   Root cause

   Capacity exhaustion

   Component

   Kubernetes API serverCilium

   Contributing factor

   Unreviewed defaults

   Detection

   A tech support engineer phoned the team to report that the customer's cluster was practically non-operational.

   Remediation

   • added CPU and memory to the control plane nodes in two steps, from 8 CPUs and 16 GB of RAM to 16 CPUs and 64 GB, until the API stabilised
   • restarted the Cilium agents in a test window agreed with the customer to reproduce the memory growth
   • deployed a FlowSchema and a PriorityLevelConfiguration named cilium-pods that limit and queue LIST requests from the Cilium service accounts to cilium.io resources
   • reduced the control plane nodes back to their original resources

   Action items

   • Make an API Priority and Fairness configuration a standard part of Palark's Kubernetes platform

   Lessons

   In a large cluster, many agents sending LIST requests at once can exhaust kube-apiserver memory, and with several API servers the crash moves from one to the next. APF can hold back such a client: a FlowSchema matches its requests and a PriorityLevelConfiguration gives them a small concurrency share and a queue, so the requests run more slowly instead of all at once. With APF enabled, the total concurrency it divides is the sum of --max-requests-inflight (default 400) and --max-mutating-requests-inflight (default 200), shared among priority levels in proportion to their assured concurrency shares, so raising one level's share lowers every other level's limit. Long-running requests such as logs and exec, and WATCH requests, are not subject to APF. The response headers X-Kubernetes-PF-FlowSchema-UID and X-Kubernetes-PF-PriorityLevel-UID show where a request landed, and the metrics Apiserver_flowcontrol_rejected_requests_total, Apiserver_current_inqueue_requests and Apiserver_flowcontrol_request_execution_seconds, with the debug endpoints /debug/api_priority_and_fairness/dump_priority_levels and /debug/api_priority_and_fairness/dump_queues, show how the queues behave.

   View case
3. IL-0165 Some LKE users in Frankfurt could not reach their kube-apiservers with kubectl after backend services failed under memory constraints

   Linode2023

   Symptoms

   At 14:11 UTC on May 25, 2023, Linode learned that part of its LKE users in the Frankfurt data center could not reach their kube-apiservers through kubectl. The status page opened at 14:27 UTC with "We are investigating reports of connectivity issues with the LKE service within the Frankfurt Datacenter".

   Control plane unresponsive

   Root cause

   Capacity exhaustion

   Component

   Kubernetes API serverContainer memory limit

   Contributing factor

   –

   Detection

   Linode was made aware of the problem at 14:11 UTC; the status page refers to reports of connectivity issues.

   Remediation

   • increased memory limits

   Action items

   • Increase resilience, redundancy and monitoring for the LKE components affected by the incident

   View case
4. IL-0068 Anonymous access to the Kubernetes API server used to plant a ClusterRoleBinding backdoor and a cryptominer DaemonSet

   Aqua Security2023

   Symptoms

   Aqua's research team Nautilus recorded the attack on one of its Kubernetes honeypots. What appeared in the cluster was a DaemonSet running the image kuberntesio/kube-controller:1.0.1 from Docker Hub on every node, using the nodes' CPU to mine Monero.

   Unauthorised access or a foreign workloadResources deleted

   Trigger

   An API server misconfigured to accept unauthenticated requests from anonymous users with privileges.

   Root cause

   Configuration change

   Component

   Kubernetes API serverRBAC and service accountsContainer registry and its credentials

   Contributing factor

   Monitoring blind spot

   Detection

   Aqua recorded the attacker's API requests on its honeypot; planted AWS access keys raised a beacon when the attacker tried to use them.

   Lessons

   Fixing an exposed API server is not enough once an attacker has been in: a ClusterRoleBinding to an attacker-created ServiceAccount keeps working after anonymous access is closed. Attackers name these objects and images after real Kubernetes components so they blend into the API audit logs and pass for legitimate deployments. Aqua recommends scanning clusters for vulnerabilities, exposed secrets and misconfigurations with a tool such as Trivy.

   View case
5. IL-0194 After a kube-apiserver restart, pod LIST requests from every pod of a Vector DaemonSet nearly tripled etcd memory and restarted all master nodes

   Palark2023

   Symptoms

   One evening, right after an ordinary restart of kube-apiserver in a customer's production cluster, etcd memory consumption rose almost threefold. The sudden growth led to a cascading restart of all the master nodes.

   Control plane unresponsiveDatabase overload

   Trigger

   An ordinary restart of kube-apiserver, after which every pod of the log collector DaemonSet sent its list requests at the same time.

   Root cause

   Capacity exhaustion

   Component

   etcdKubernetes API serverTelemetry agent

   Contributing factor

   Unknown limitUnreviewed defaults

   Detection

   Memory graphs showed etcd memory on the master nodes almost tripling after a kube-apiserver restart; the audit logs then pointed to one DaemonSet's list requests.

   Remediation

   • created a PriorityLevelConfiguration named limit-list-custom that queues requests (assuredConcurrencyShares 5, 16 queues, handSize 4, queueLengthLimit 50)
   • created a FlowSchema that routes list and get requests on pods from the log collector's service account to that priority level

   Action items

   • Submit a pull request adding resource version support to the Kubernetes Rust library
   • Open a pull request to the Vector log collector to address the startup list requests
   • Apply the same flow-control approach to a similar problem found with the Cilium CNI

   Lessons

   A field selector reduces what kube-apiserver returns, not what it reads from etcd: listing the pods of one node still reads every pod in the cluster. A DaemonSet whose pods all list at startup multiplies that by the node count each time kube-apiserver restarts. Where exact freshness is not needed, resourceVersion=0 lets kube-apiserver answer from its cache; where the client cannot set it, API Priority and Fairness can queue the requests so they reach etcd gradually.

   View case

By root cause9