Cases8 cases

3 to 7 of 8

1. IL-0141 gke-metadata-server versions 0.4.272 to 0.4.280 flooded Cloud Logging with "Unable to sync sandbox" debug logs on GKE clusters with Workload Identity, using up logging quota and adding billable usage

   Google Cloud2023

   Symptoms

   Google Kubernetes Engine (GKE) customers with Workload Identity enabled saw a high logging rate. Debug log lines containing the string "Unable to sync sandbox" were ingested into Cloud Logging, used up the Cloud Logging ingestion quota and, beyond the free monthly allotment, produced extra billable usage. Workload Identity is opt-in on GKE Standard and always enabled on GKE Autopilot.

   Log floodHigh latency

   Trigger

   GKE clusters ran gke-metadata-server versions 0.4.272 to 0.4.280, which contained an incorrect configuration that wrote "Unable to sync sandbox" debug logs at a high rate.

   Root cause

   Software defect

   Component

   Credential broker for pods

   Contributing factor

   –

   Detection

   The first status update was posted at 13:32 US/Pacific on 2023-11-02, and the updates told customers how to check a cluster from the version annotation on the gke-metadata-server DaemonSet.

   Remediation

   • rolled out a fix across the GKE fleet so the excess logs were no longer ingested into Cloud Logging
   • offered GKE Rapid channel customers a control plane upgrade to 1.28.2-gke.1157000 or above, or 1.27.7-gke.1038000 or above, as a workaround

   View case
2. IL-0530 Virtual router controllers in us-central1 ran out of memory in a loop, cutting networking for new and migrated VMs

   Google Cloud2023

   Symptoms

   On 5 October 2023 in us-central1, newly created VMs waited a long time before their networking worked, and VMs that had been live migrated lost connectivity for a long time after the migration. Networking telemetry showed cross-region packet loss to and from us-central1 from 03:08 US/Pacific. Health check state of VMs reached load balancers late.

   Packet lossNetwork unreachableOOMKilledStale endpoint routingCluster or node provisioning stall

   Trigger

   A management plane behaviour change, meant to decouple the processing of API updates to Instance Groups and Network Endpoint Groups used as load balancer backends, was deployed in us-central1 after rolling out to several other regions without incident.

   Root cause

   Faulty deployment

   Component

   Network control plane (SDN)Cloud load balancer

   Contributing factor

   Excessive blast radiusNo alert on limit

   Detection

   Engineers were alerted to slowness in the virtual network control plane in us-central1 at 21:45 US/Pacific on 4 October, before the packet loss started, and to controller failures from memory exhaustion at 02:11 on 5 October. Networking telemetry showed cross-region packet loss from 03:08.

   Remediation

   • Increased the memory allocated to the virtual network router controllers
   • Rolled back the management plane change in us-central1
   • Restarted by hand the routers that were still not accepting updates

   Action items

   • Alert proactively on memory risks and unexpected memory increases
   • Refactor the deployment configuration so engineers can reallocate memory much faster
   • Re-evaluate existing practices and add safety mechanisms for API reconciliation
   • Make management plane changes more visible across teams so the changes can be correlated faster
   • Adjust the deployment footprint so a regional API change is less likely to exhaust memory across the region at once
   • Optimise memory use in the traffic routers and their controllers

   View case
3. IL-0142 GKE clusters on 1.24 and later wrote unexpected fluentbit error entries such as "Failed to get record: decoder: failed to decode payload: EOF" that used up the Logging API write quota for some customers

   Google Cloud2023

   Symptoms

   From 07:13 US/Pacific on 6 July 2023, GKE customers saw unexpected additional error and warning entries in their cluster logs, written to the Cloud Logging project that holds the clusters. Standard and Autopilot clusters at GKE 1.24.x and later were affected.

   Log flood

   Trigger

   Google rolled out a new version of the GKE component that processes logs (preliminary analysis).

   Root cause

   Faulty deployment

   Component

   Telemetry agent

   Contributing factor

   –

   Remediation

   • gave customers a workaround: request a Logging API quota increase, or apply a log exclusion for the fluentbit-gke messages
   • rolled out a fix gradually to the affected clusters between 10 and 18 July 2023

   Lessons

   Log messages written by a managed logging agent count against the customer's own Logging API write quota and logging bill, so a faulty agent release can drop a customer's other logs. A log exclusion on the agent's container name and the message patterns keeps the noise out of storage until a fix arrives.

   View case
4. IL-0139 Pods on GKE clusters with Calico network policy got stuck Terminating or Pending after upgrading to 1.22 or later because of a race condition in the Calico component

   Google Cloud2022

   Symptoms

   After upgrading to GKE 1.22 or later, clusters that use Calico network policy could have pods stuck in Terminating or Pending. Google Cloud posted the incident at 16:01 US/Pacific on 2022-09-15 as pods stuck terminating; from the 2022-09-21 update the description also named pods stuck Pending.

   Pods stuck in TerminatingPods Pending

   Trigger

   Upgrading a GKE cluster that uses Calico network policy to version 1.22 or later.

   Root cause

   Software defect

   Component

   CalicokubeletPod termination

   Contributing factor

   –

   Remediation

   • released a fix in GKE 1.24.4-gke.800 and made it available in 1.23 and 1.22
   • offered an internal patch through Google Cloud Support for affected customers
   • advised customers to restart the kubelet and calico-node to get stuck pods unstuck
   • advised customers without a strong need for Calico to disable Calico network policy

   Action items

   • Upgrade clusters on the Rapid, Regular and Stable release channels running 1.22 or 1.23 to the fixed version automatically

   Lessons

   A race in the Calico component can leave pods stuck Terminating, with FailedKillPod and KillPodSandbox errors that point at the network plugin, not at the workload. Restarting the kubelet and calico-node frees the stuck pods but does not remove the race; the fixed GKE version or Google's internal patch does. Clusters that use the cluster autoscaler are more likely to hit it.

   View case
5. IL-0140 The GKE API in us-central1 returned errors and high latency for 3 hours 19 minutes after an unexpected rise in backend traffic that client retries made worse

   Google Cloud2022

   Symptoms

   From 03:03 US/Pacific on 14 June 2022, calls to the GKE API in us-central1 through gcloud and the Google Cloud Console were slow and returned errors. Retrying sometimes succeeded.

   High error rateHigh latencyControl plane lag

   Trigger

   An unexpected increase in backend traffic to the GKE API in us-central1.

   Root cause

   Capacity exhaustion

   Component

   GKE control plane

   Contributing factor

   Retry storm

   Remediation

   • scaled up the GKE control plane to handle the increased traffic

   Lessons

   Client retries against an overloaded management API add to the overload. When the cloud provider's control plane is degraded, running workloads may be fine while cluster operations fail, and the status page offered no workaround beyond advising customers to use another region if they could.

   View case

By root cause5