Cases46 cases

16 to 20 of 46

1. IL-0077 EKS cluster went read-only for over seven hours after Kyverno ephemeralreports filled the etcd database past its size limit

   Marcin Cuber2024sev1

   Symptoms

   A production EKS cluster on Kubernetes 1.30 went into read-only mode: the API server rejected every request that would change an object, such as creating pods or scaling deployments, and deleting objects to free space did not work either.

   Control plane unresponsiveControl plane lag

   Trigger

   Kyverno 1.12 kept creating ephemeralreports objects that piled up in etcd until the database passed EKS's size limit.

   Root cause

   Software defect

   Component

   etcdKubernetes API serverPolicy engine

   Contributing factor

   No alert on limitLimited debug access

   Detection

   The cluster rejected writes; the audit logs carried the "etcdserver: mvcc: database space exceeded" error.

   Remediation

   • AWS raised the etcd storage limit from 8/10GB to 12GB to unlock writes
   • an AWS engineer deleted the leaked Kyverno ephemeralreports objects directly from etcd

   Action items

   • Monitor the etcd database size with apiserver_storage_size_bytes and never let it reach 8GB
   • Upgrade Kyverno 1.12 to 1.12.4, which fixes the ephemeralreports pile-up, or disable reporting for admission events and tune --aggregationWorkers

   Lessons

   On EKS, once etcd hits its size limit the cluster becomes read-only and only AWS can clean it up, so cleanup scripts must run before the limit is reached. Watch the etcd database size and object counts per resource, especially for controllers that create report objects.

   View case
2. IL-0161 A faulty earlier API update made new LKE cluster provisioning in Toronto time out for three days, leaving clusters with no node pools or the wrong number of nodes

   Linode2024

   Symptoms

   At about 14:05 UTC on 22 April 2024, Linode's team noticed that new LKE clusters were failing to provision in the Toronto data center.

   Cluster or node provisioning stallIntermittent timeouts

   Trigger

   A previous update to the API did not work as intended and caused timeouts when new LKE clusters were provisioned.

   Root cause

   Software defect

   Component

   Node pool

   Contributing factor

   –

   Detection

   Linode's team noticed new LKE clusters failing to provision in the Toronto data center at about 14:05 UTC on 22 April 2024.

   Remediation

   • disabled the affected section of Toronto infrastructure from provisioning new LKE clusters
   • implemented a fix for the faulty API update
   • put the disabled infrastructure back into use after the fix

   Lessons

   The fault was in an earlier API change, not in the infrastructure that showed it, but most failures concentrated on one backend resource in one data center. Taking that resource out of provisioning limited the impact while the root cause was found, which took three days.

   View case
3. IL-0173 cert-manager's preferredChain logic picked the Let's Encrypt chain through the expired DST Root CA X3, so clients built on an old BoringSSL could not connect to a development server

   Devsisters2024

   Symptoms

   On 13 February 2024, when Devsisters' infrastructure cell came back from the Lunar New Year holiday, it received a request to look into a development server that could not be reached. No one had made a manual change over the holiday.

   TLS certificate errors

   Trigger

   On 8 February 2024, in the first phase of retiring the DST Root CA X3 cross-sign, Let's Encrypt made the ISRG Root X1 chain its default chain and began offering the DST Root CA X3 chain as the alternate.

   Root cause

   Software defect

   Component

   cert-manager

   Contributing factor

   –

   Detection

   On 13 February, after the Lunar New Year holiday, a request came in to the infrastructure cell that the development server could not be reached; while inspecting the certificate chain the server presented, the team found it serving the chain through the expired DST Root CA X3.

   Remediation

   • submitted a pull request fixing cert-manager's preferred chain selection, which was merged

   Lessons

   A latent bug can stay hidden until an upstream provider changes a default. cert-manager's preferredChain matching gave the intended result while DST Root CA X3 was Let's Encrypt's default chain and the wrong one after the default changed. An expired root at the top of a served chain is harmless to most TLS clients, which fall back to the trusted ISRG Root X1 below it, but fatal to an implementation such as an old BoringSSL that checks the trust anchor's validity and insists on verifying the topmost certificate. ACME clients choose the preferred chain with different heuristics, so the same preferredChain value can behave differently between Certbot and cert-manager.

   View case
4. IL-0065 Any pod could take over the cluster by rewriting Argo CD's unauthenticated Redis cache (CVE-2024-31989)

   Cycode2024

   Symptoms

   This is a vulnerability report, not an outage, so nothing failed visibly. What an engineer can check is whether the Argo CD Redis server accepts connections without a password from a pod in another namespace, on port 6379 at the address kube-dns returns. In the default installation it did.

   Unauthorised access or a foreign workload

   Trigger

   Cycode Labs installed Argo CD with the default configuration on a new Amazon EKS cluster and connected to its Redis server from a low-privilege pod in another namespace, simulating a compromised pod.

   Root cause

   Software defect

   Component

   Argo CD

   Contributing factor

   Unreviewed defaultsExcessive blast radiusCredential in configuration

   Detection

   Found by Cycode Labs security research, not by an operator.

   Remediation

   • Argo CD released 2.11.1, 2.10.10, 2.9.15 and 2.8.19 with an initialized password for the Redis server

   Lessons

   The hash that checks the cached manifests is computed without a secret key, so anyone who can write to Redis can recompute it and make Argo CD deploy an altered manifest. Cycode recommends upgrading to the fixed versions, checking that the argocd-redis-network-policy NetworkPolicy exists and that a CNI enforces it, using controller-based secret management instead of injecting secrets through Argo CD plugins so secrets never reach Redis, and running Argo CD on a dedicated cluster.

   View case
5. IL-0141 gke-metadata-server versions 0.4.272 to 0.4.280 flooded Cloud Logging with "Unable to sync sandbox" debug logs on GKE clusters with Workload Identity, using up logging quota and adding billable usage

   Google Cloud2023

   Symptoms

   Google Kubernetes Engine (GKE) customers with Workload Identity enabled saw a high logging rate. Debug log lines containing the string "Unable to sync sandbox" were ingested into Cloud Logging, used up the Cloud Logging ingestion quota and, beyond the free monthly allotment, produced extra billable usage. Workload Identity is opt-in on GKE Standard and always enabled on GKE Autopilot.

   Log floodHigh latency

   Trigger

   GKE clusters ran gke-metadata-server versions 0.4.272 to 0.4.280, which contained an incorrect configuration that wrote "Unable to sync sandbox" debug logs at a high rate.

   Root cause

   Software defect

   Component

   Credential broker for pods

   Contributing factor

   –

   Detection

   The first status update was posted at 13:32 US/Pacific on 2023-11-02, and the updates told customers how to check a cluster from the version annotation on the gke-metadata-server DaemonSet.

   Remediation

   • rolled out a fix across the GKE fleet so the excess logs were no longer ingested into Cloud Logging
   • offered GKE Rapid channel customers a control plane upgrade to 1.28.2-gke.1157000 or above, or 1.27.7-gke.1038000 or above, as a workaround

   View case

By root cause1