Incident Library
Applied filters Root causeSoftware defectComponentCalicoClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0139 Pods on GKE clusters with Calico network policy got stuck Terminating or Pending after upgrading to 1.22 or later because of a race condition in the Calico component
    Google Cloud2022
    Symptoms

    After upgrading to GKE 1.22 or later, clusters that use Calico network policy could have pods stuck in Terminating or Pending. Google Cloud posted the incident at 16:01 US/Pacific on 2022-09-15 as pods stuck terminating; from the 2022-09-21 update the description also named pods stuck Pending.

    Pods stuck in TerminatingPods Pending
    Trigger

    Upgrading a GKE cluster that uses Calico network policy to version 1.22 or later.

    Root cause
    Software defect
    Component
    CalicokubeletPod termination
    Contributing factor
    –
    Remediation
    • released a fix in GKE 1.24.4-gke.800 and made it available in 1.23 and 1.22
    • offered an internal patch through Google Cloud Support for affected customers
    • advised customers to restart the kubelet and calico-node to get stuck pods unstuck
    • advised customers without a strong need for Calico to disable Calico network policy
    Action items
    • Upgrade clusters on the Rapid, Regular and Stable release channels running 1.22 or 1.23 to the fixed version automatically
    Lessons

    A race in the Calico component can leave pods stuck Terminating, with FailedKillPod and KillPodSandbox errors that point at the network plugin, not at the workload. Restarting the kubelet and calico-node frees the stuck pods but does not remove the race; the fixed GKE version or Google's internal patch does. Clusters that use the cluster autoscaler are more likely to hit it.

    View case
Check your cluster for this

This case involves Calico and kubelet. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Software defect1 case

Component
CalicokubeletPod termination
Contributing factor
–
Organisation
Google Cloud
  1. IL-0139 Pods on GKE clusters with Calico network policy got stuck Terminating or Pending after upgrading to 1.22 or later because of a race condition in the Calico component
    Google Cloud2022