Cases6 cases

5 to 6 of 6

1. IL-0139 Pods on GKE clusters with Calico network policy got stuck Terminating or Pending after upgrading to 1.22 or later because of a race condition in the Calico component

   Google Cloud2022

   Symptoms

   After upgrading to GKE 1.22 or later, clusters that use Calico network policy could have pods stuck in Terminating or Pending. Google Cloud posted the incident at 16:01 US/Pacific on 2022-09-15 as pods stuck terminating; from the 2022-09-21 update the description also named pods stuck Pending.

   Pods stuck in TerminatingPods Pending

   Trigger

   Upgrading a GKE cluster that uses Calico network policy to version 1.22 or later.

   Root cause

   Software defect

   Component

   CalicokubeletPod termination

   Contributing factor

   –

   Remediation

   • released a fix in GKE 1.24.4-gke.800 and made it available in 1.23 and 1.22
   • offered an internal patch through Google Cloud Support for affected customers
   • advised customers to restart the kubelet and calico-node to get stuck pods unstuck
   • advised customers without a strong need for Calico to disable Calico network policy

   Action items

   • Upgrade clusters on the Rapid, Regular and Stable release channels running 1.22 or 1.23 to the fixed version automatically

   Lessons

   A race in the Calico component can leave pods stuck Terminating, with FailedKillPod and KillPodSandbox errors that point at the network plugin, not at the workload. Restarting the kubelet and calico-node frees the stuck pods but does not remove the race; the fixed GKE version or Google's internal patch does. Clusters that use the cluster autoscaler are more likely to hit it.

   View case
2. IL-0086 A deployment meant for a CircleCI test Kubernetes cluster also removed Traefik and Calico resources from the production cluster, taking down the CircleCI UI and runner.circleci.com

   CircleCI2021

   Symptoms

   At 15:35 EST on 11 March 2021 the CircleCI web UI and runner.circleci.com became unavailable. The API, which uses a separate ingress point, stayed reachable, and builds kept running.

   Service outageResources deleted

   Trigger

   An update deployed to a test Kubernetes cluster also removed Traefik and Calico resources from CircleCI's primary production cluster.

   Root cause

   Faulty deployment

   Component

   Ingress controllerCalico

   Contributing factor

   Excessive blast radiusNo runbookMonitoring blind spot

   Detection

   CircleCI engineers were alerted and began investigating. The status page posted its first notice at 20:45 UTC.

   Remediation

   • restored Traefik in the production cluster
   • restored Calico in the production cluster, after which pods served customer requests again

   Action items

   • Improve network-level documentation and runbooks to cover every layer of the Kubernetes networks
   • Improve monitoring and alerting for the network infrastructure
   • Add guardrails that isolate the production and staging clusters so they cannot interfere with each other

   Lessons

   A deployment path that can reach both a test cluster and production can remove shared networking components from production. When the ingress controller and the CNI are both affected, restoring the ingress alone is not enough: engineers had to find the link between Calico and Traefik and restore Calico before pods served traffic again. Time went to DNS and AWS red herrings because nothing linked the outage to the test deployment; CircleCI's answer is better network runbooks, network alerting and guardrails between clusters.

   View case

By root cause6