Cases2 cases

1 to 2 of 2

1. IL-0077 EKS cluster went read-only for over seven hours after Kyverno ephemeralreports filled the etcd database past its size limit

   Marcin Cuber2024sev1

   Symptoms

   A production EKS cluster on Kubernetes 1.30 went into read-only mode: the API server rejected every request that would change an object, such as creating pods or scaling deployments, and deleting objects to free space did not work either.

   Control plane unresponsiveControl plane lag

   Trigger

   Kyverno 1.12 kept creating ephemeralreports objects that piled up in etcd until the database passed EKS's size limit.

   Root cause

   Software defect

   Component

   etcdKubernetes API serverPolicy engine

   Contributing factor

   No alert on limitLimited debug access

   Detection

   The cluster rejected writes; the audit logs carried the "etcdserver: mvcc: database space exceeded" error.

   Remediation

   • AWS raised the etcd storage limit from 8/10GB to 12GB to unlock writes
   • an AWS engineer deleted the leaked Kyverno ephemeralreports objects directly from etcd

   Action items

   • Monitor the etcd database size with apiserver_storage_size_bytes and never let it reach 8GB
   • Upgrade Kyverno 1.12 to 1.12.4, which fixes the ephemeralreports pile-up, or disable reporting for admission events and tune --aggregationWorkers

   Lessons

   On EKS, once etcd hits its size limit the cluster becomes read-only and only AWS can clean it up, so cleanup scripts must run before the limit is reached. Watch the etcd database size and object counts per resource, especially for controllers that create report objects.

   View case
2. IL-0504 mybinder.org launches failed for about five hours as undeleted user pods piled up and filled the cluster

   Binder Project2018

   Symptoms

   On Saturday 8 July 2018 at about 18:30 CEST, the launch success rate of mybinder.org dropped to zero and stayed there. Suspicious behaviour was first reported on Gitter at 22:48, more than four hours later.

   Service outagePods PendingStale or inconsistent API reads

   Root cause

   Software defect

   Component

   JupyterHub

   Contributing factor

   No alert on limit

   Detection

   Suspicious behaviour was reported on Gitter at 22:48 CEST, more than four hours after launch success dropped to zero; no downtime notifications were set up.

   Remediation

   • Deleted user pods older than four hours with the delete-pods.py script
   • Culled pods of the most-used images, such as ipython-in-depth, more aggressively
   • Restarted the Hub pod after about 600 pods were culled, which brought launches back to normal
   • Cordoned some of the extra nodes so the autoscaler could reclaim them
   • Ran manual max-age culling the next morning, letting the cluster scale back to 3 nodes

   Action items

   • Set up notifications of downtime
   • Automate cluster scale-down
   • Document the downtime recovery steps so every team member knows them
   • Fix KubeSpawner's failure to delete pods when the pod reflector fails
   • Make KubeSpawner more robust to pod reflector failure
   • Refactor the event reflector into a single reflector instead of one per pod
   • Detect a likely unhealthy Hub state and restart the Hub without manual intervention

   Lessons

   The delete-pods.py script was useful for bulk-deleting old pods, and once the problem was noticed the cluster was healthy within a few minutes. The outage lasted five hours before anyone noticed because no downtime notifications existed, and scaling the cluster back down still took several manual steps of cordoning and draining nodes.

   View case

By root cause1