Incident Library
Applied filters ComponentKubernetes controller managerFactorExcessive blast radiusClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0038 kubelet rollout hit a Kubernetes bug and left every node in a cluster NotReady, triggering mass pod eviction
    Oath2018
    Symptoms

    During a rolling update of a kubelet change in one of Oath's on-premises production clusters, nodes started going NotReady, and soon all nodes in the cluster were NotReady.

    Node NotReadyPod evictionPods Pending
    Trigger

    A change to kubelet was rolled out across the nodes of one cluster; on restart, kubelet cleans up transient cgroup files, and a Kubernetes bug in that step left kubelet stuck instead of starting.

    Root cause
    Software defect
    Component
    kubeletKubernetes controller manager
    Contributing factor
    No health probesExcessive blast radius
    Detection

    Nodes reported NotReady and the eviction graphs climbed while the kubelet rollout was in progress.

    Remediation
    • failed the affected data center over so users were routed to the next available data center
    Action items
    • Run a functional test of kubelet after each upgrade batch and certify it before the next 30% of nodes are updated
    Lessons

    Check kubelet after upgrading it before moving to the next batch; a simple curl against the kubelet API would have caught this. The node controller's eviction throttling and full disruption mode, which the speakers note is not well documented, kept a cluster-wide kubelet failure from evicting every pod.

    View case
Check your cluster for this

This case involves Kubernetes controller manager and kubelet. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Software defect1 case

Component
Kubernetes controller managerkubelet
Contributing factor
Excessive blast radiusNo health probes
Organisation
Oath
  1. IL-0038 kubelet rollout hit a Kubernetes bug and left every node in a cluster NotReady, triggering mass pod eviction
    Oath2018