Incident Library
Applied filters Root causeSoftware defectComponentkubeletSymptomPods PendingClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0139 Pods on GKE clusters with Calico network policy got stuck Terminating or Pending after upgrading to 1.22 or later because of a race condition in the Calico component
    Google Cloud2022
    Symptoms

    After upgrading to GKE 1.22 or later, clusters that use Calico network policy could have pods stuck in Terminating or Pending. Google Cloud posted the incident at 16:01 US/Pacific on 2022-09-15 as pods stuck terminating; from the 2022-09-21 update the description also named pods stuck Pending.

    Pods stuck in TerminatingPods Pending
    Trigger

    Upgrading a GKE cluster that uses Calico network policy to version 1.22 or later.

    Root cause
    Software defect
    Component
    CalicokubeletPod termination
    Contributing factor
    –
    Remediation
    • released a fix in GKE 1.24.4-gke.800 and made it available in 1.23 and 1.22
    • offered an internal patch through Google Cloud Support for affected customers
    • advised customers to restart the kubelet and calico-node to get stuck pods unstuck
    • advised customers without a strong need for Calico to disable Calico network policy
    Action items
    • Upgrade clusters on the Rapid, Regular and Stable release channels running 1.22 or 1.23 to the fixed version automatically
    Lessons

    A race in the Calico component can leave pods stuck Terminating, with FailedKillPod and KillPodSandbox errors that point at the network plugin, not at the workload. Restarting the kubelet and calico-node frees the stuck pods but does not remove the race; the fixed GKE version or Google's internal patch does. Clusters that use the cluster autoscaler are more likely to hit it.

    View case
  2. IL-0038 kubelet rollout hit a Kubernetes bug and left every node in a cluster NotReady, triggering mass pod eviction
    Oath2018
    Symptoms

    During a rolling update of a kubelet change in one of Oath's on-premises production clusters, nodes started going NotReady, and soon all nodes in the cluster were NotReady.

    Node NotReadyPod evictionPods Pending
    Trigger

    A change to kubelet was rolled out across the nodes of one cluster; on restart, kubelet cleans up transient cgroup files, and a Kubernetes bug in that step left kubelet stuck instead of starting.

    Root cause
    Software defect
    Component
    kubeletKubernetes controller manager
    Contributing factor
    No health probesExcessive blast radius
    Detection

    Nodes reported NotReady and the eviction graphs climbed while the kubelet rollout was in progress.

    Remediation
    • failed the affected data center over so users were routed to the next available data center
    Action items
    • Run a functional test of kubelet after each upgrade batch and certify it before the next 30% of nodes are updated
    Lessons

    Check kubelet after upgrading it before moving to the next batch; a simple curl against the kubelet API would have caught this. The node controller's eviction throttling and full disruption mode, which the speakers note is not well documented, kept a cluster-wide kubelet failure from evicting every pod.

    View case
Check your cluster for this

These 2 cases involve kubelet and Calico. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Software defect2 cases

Component
kubelet2Calico1Kubernetes controller manager1Pod termination1
Contributing factor
Excessive blast radius1No health probes1
Organisation
Google Cloud1Oath1
  1. IL-0139 Pods on GKE clusters with Calico network policy got stuck Terminating or Pending after upgrading to 1.22 or later because of a race condition in the Calico component
    Google Cloud2022
  2. IL-0038 kubelet rollout hit a Kubernetes bug and left every node in a cluster NotReady, triggering mass pod eviction
    Oath2018