Incident Library
Applied filters ComponentKubernetes API serverSymptomDatabase overloadFactorUnknown limitClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0194 After a kube-apiserver restart, pod LIST requests from every pod of a Vector DaemonSet nearly tripled etcd memory and restarted all master nodes
    Palark2023
    Symptoms

    One evening, right after an ordinary restart of kube-apiserver in a customer's production cluster, etcd memory consumption rose almost threefold. The sudden growth led to a cascading restart of all the master nodes.

    Control plane unresponsiveDatabase overload
    Trigger

    An ordinary restart of kube-apiserver, after which every pod of the log collector DaemonSet sent its list requests at the same time.

    Root cause
    Capacity exhaustion
    Component
    etcdKubernetes API serverTelemetry agent
    Contributing factor
    Unknown limitUnreviewed defaults
    Detection

    Memory graphs showed etcd memory on the master nodes almost tripling after a kube-apiserver restart; the audit logs then pointed to one DaemonSet's list requests.

    Remediation
    • created a PriorityLevelConfiguration named limit-list-custom that queues requests (assuredConcurrencyShares 5, 16 queues, handSize 4, queueLengthLimit 50)
    • created a FlowSchema that routes list and get requests on pods from the log collector's service account to that priority level
    Action items
    • Submit a pull request adding resource version support to the Kubernetes Rust library
    • Open a pull request to the Vector log collector to address the startup list requests
    • Apply the same flow-control approach to a similar problem found with the Cilium CNI
    Lessons

    A field selector reduces what kube-apiserver returns, not what it reads from etcd: listing the pods of one node still reads every pod in the cluster. A DaemonSet whose pods all list at startup multiplies that by the node count each time kube-apiserver restarts. Where exact freshness is not needed, resourceVersion=0 lets kube-apiserver answer from its cache; where the client cannot set it, API Priority and Fairness can queue the requests so they reach etcd gradually.

    View case
Check your cluster for this

This case involves etcd and Kubernetes API server. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Capacity exhaustion1 case

Component
etcdKubernetes API serverTelemetry agent
Contributing factor
Unreviewed defaultsUnknown limit
Organisation
Palark
  1. IL-0194 After a kube-apiserver restart, pod LIST requests from every pod of a Vector DaemonSet nearly tripled etcd memory and restarted all master nodes
    Palark2023