IL-0139 Google Cloud · 2022-09-15kubernetesgcpnetworking
A race condition introduced in the Calico component affected GKE 1.22, 1.23 and 1.24 before 1.24.4-gke.800. On clusters with Calico network policy the race left pods stuck Terminating or Pending; when terminating pods, the Calico CNI plugin failed to tear down the pod network with an Unauthorized error on getting ClusterInformation. GKE worked with the Calico project on a fix.
The fix shipped first in GKE 1.24.4-gke.800 and was then made available in 1.23 and 1.22; clusters on the Rapid, Regular and Stable release channels running 1.22 or 1.23 were to upgrade automatically over the following weeks. Until then, the recommended action was to upgrade manually to 1.24.4-gke.800 or later or to ask Google Cloud Support for an internal patch; restarting the kubelet and calico-node got stuck pods moving, and in the first update disabling Calico network policy was offered for customers without a strong need for Calico. Google Cloud reported the issue resolved for all affected users as of 13:45 US/Pacific on 2022-09-29.
After upgrading to GKE 1.22 or later, clusters that use Calico network policy could have pods stuck in Terminating or Pending. Google Cloud posted the incident at 16:01 US/Pacific on 2022-09-15 as pods stuck terminating; from the 2022-09-21 update the description also named pods stuck Pending.
When terminating pods, the Calico CNI plugin reports: "Warning FailedKillPod 36m (x389 over 121m) kubelet error killing pod: failed to "KillPodSandbox" for "af9ab8f9-d6d6-4828-9b8c-a58441dd1f86" with KillPodSandboxError: "rpc error: code = Unknown desc = networkPlugin cni failed to teardown pod "myclient-pod-6474c76996" network: error getting ClusterInformation: connection is unauthorized: Unauthorized".
The affected versions were all 1.22 and all 1.23 GKE versions, and 1.24 versions before 1.24.4-gke.800. Only a small number of GKE clusters actually experienced stuck pods, and use of the cluster autoscaler increased the chance of hitting the race condition.
Root cause Software defect
–
A race in the Calico component can leave pods stuck Terminating, with FailedKillPod and KillPodSandbox errors that point at the network plugin, not at the workload. Restarting the kubelet and calico-node frees the stuck pods but does not remove the race; the fixed GKE version or Google's internal patch does. Clusters that use the cluster autoscaler are more likely to hit it.
IL-0112 The daily node rollover put DFDS's EKS nodes on an AMI later recalled for a kernel issue, causing "PLEG is not healthy" NodeNotReady flaps and pod network timeoutsDFDS · 2022same root cause · shared kubelet
IL-0012 CPU limits and an unpatched kernel CFS quota bug throttled multi-threaded services into stalls, failed health checks and timeoutsOmio · 2020same root cause · shared kubelet
IL-0062 Intermittent 502 errors behind a GKE Ingress because pods shut down before the load balancer stopped sending them trafficRavelin · 2019same root cause · shared Pod termination
IL-0074 All AirMap platform services down as worker nodes went NotReady and pods crash-looped on network and DNS failuresAirMap · 2018same root cause · shared kubelet
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.
same root cause · shared kubelet