Incident Library
Applied filters ComponentPolicy engineFactorNo alert on limitClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0077 EKS cluster went read-only for over seven hours after Kyverno ephemeralreports filled the etcd database past its size limit
    Marcin Cuber2024sev1
    Symptoms

    A production EKS cluster on Kubernetes 1.30 went into read-only mode: the API server rejected every request that would change an object, such as creating pods or scaling deployments, and deleting objects to free space did not work either.

    Control plane unresponsiveControl plane lag
    Trigger

    Kyverno 1.12 kept creating ephemeralreports objects that piled up in etcd until the database passed EKS's size limit.

    Root cause
    Software defect
    Component
    etcdKubernetes API serverPolicy engine
    Contributing factor
    No alert on limitLimited debug access
    Detection

    The cluster rejected writes; the audit logs carried the "etcdserver: mvcc: database space exceeded" error.

    Remediation
    • AWS raised the etcd storage limit from 8/10GB to 12GB to unlock writes
    • an AWS engineer deleted the leaked Kyverno ephemeralreports objects directly from etcd
    Action items
    • Monitor the etcd database size with apiserver_storage_size_bytes and never let it reach 8GB
    • Upgrade Kyverno 1.12 to 1.12.4, which fixes the ephemeralreports pile-up, or disable reporting for admission events and tune --aggregationWorkers
    Lessons

    On EKS, once etcd hits its size limit the cluster becomes read-only and only AWS can clean it up, so cleanup scripts must run before the limit is reached. Watch the etcd database size and object counts per resource, especially for controllers that create report objects.

    View case
Check your cluster for this

This case involves etcd and Kubernetes API server. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Software defect1 case

Component
etcdKubernetes API serverPolicy engine
Contributing factor
Limited debug accessNo alert on limit
Organisation
Marcin Cuber
  1. IL-0077 EKS cluster went read-only for over seven hours after Kyverno ephemeralreports filled the etcd database past its size limit
    Marcin Cuber2024