Cases2 cases

1 to 2 of 2

1. IL-0139 Pods on GKE clusters with Calico network policy got stuck Terminating or Pending after upgrading to 1.22 or later because of a race condition in the Calico component

   Google Cloud2022

   Symptoms

   After upgrading to GKE 1.22 or later, clusters that use Calico network policy could have pods stuck in Terminating or Pending. Google Cloud posted the incident at 16:01 US/Pacific on 2022-09-15 as pods stuck terminating; from the 2022-09-21 update the description also named pods stuck Pending.

   Pods stuck in TerminatingPods Pending

   Trigger

   Upgrading a GKE cluster that uses Calico network policy to version 1.22 or later.

   Root cause

   Software defect

   Component

   CalicokubeletPod termination

   Contributing factor

   –

   Remediation

   • released a fix in GKE 1.24.4-gke.800 and made it available in 1.23 and 1.22
   • offered an internal patch through Google Cloud Support for affected customers
   • advised customers to restart the kubelet and calico-node to get stuck pods unstuck
   • advised customers without a strong need for Calico to disable Calico network policy

   Action items

   • Upgrade clusters on the Rapid, Regular and Stable release channels running 1.22 or 1.23 to the fixed version automatically

   Lessons

   A race in the Calico component can leave pods stuck Terminating, with FailedKillPod and KillPodSandbox errors that point at the network plugin, not at the workload. Restarting the kubelet and calico-node frees the stuck pods but does not remove the race; the fixed GKE version or Google's internal patch does. Clusters that use the cluster autoscaler are more likely to hit it.

   View case
2. IL-0401 Prow jobs on the k8s-prow-build GKE cluster failed and merges stalled as nodes hit IOPS throttling and kernel panics

   Kubernetes project2019

   Symptoms

   On 30 May 2019, in the run-up to the Kubernetes 1.15 code freeze, Prow test jobs in the k8s-prow-build cluster failed in surprising ways or were aborted partway through. Jobs were reported to GitHub as failed while Deck still showed them as running, because pods exited without uploading `finished.json`, the file most Prow tooling uses to tell a job's state.

   Node NotReadyNode kernel panicPods stuck in TerminatingThroughput degradation

   Trigger

   A spike in test load before the Kubernetes 1.15 code freeze pushed disk I/O on the build nodes far beyond the IOPS that GCP allows for 250 GB disks.

   Root cause

   Capacity exhaustion

   Component

   Cloud block storageNode root diskkubeletCI system

   Contributing factor

   Monitoring blind spotIgnored early signal

   Detection

   Several users reported jobs that GitHub showed as failed but Deck showed as still running, and the first report came on Slack at 13:26. Alerting for these failures did not exist or did not work.

   Remediation

   • Requested and received additional disk quota
   • Brought up 55 nodes with 500 GB disks in a new node pool
   • Migrated all nodes from the 250 GB disk pool to the 500 GB disk pool by draining old nodes, finishing at 23:55

   Action items

   • Create or fix monitoring so the oncall is paged when tide stops doing things
   • Set a process and guidelines for adding new members to the oncall rotation, with preparation and extra support

   Lessons

   Prow metrics were not recorded before the incident, which made it hard to understand, and alerting for these failures did not exist or did not work. After the cause was found and a fix devised, nothing happened for another two hours. The team counts it as luck that someone who recognised IOPS as the problem happened to be around to look.

   View case

By root cause2