Incident Library

IL-0530 Google Cloud · 2023-10-05gcpnetworkingkubernetes

Virtual router controllers in us-central1 ran out of memory in a loop, cutting networking for new and migrated VMs

Google Cloud had been slowly rolling out a management plane change that decoupled the processing of API updates to Instance Groups and Network Endpoint Groups used as load balancer backends. The change ran in several regions without problems, but in us-central1 the large workload sizes made the memory use of the control plane for the virtual network routers grow unexpectedly. The controllers ran out of memory, were restarted automatically and ran out of memory again, over and over. Virtual routers and their controllers sit in separate zonal failure domains, but the change acted through a regional API, so every virtual router controller in the region came under memory pressure at the same time. Without working controllers, the routers were not updated with new VMs, new locations of migrated VMs, dynamic routes or load balancer backend health. Intra-region and cross-region traffic use separate sets of virtual routers with their own control plane, and the cross-region routers were affected first and longer.

Engineers were alerted to intermittent control plane slowness at 21:45 US/Pacific on 4 October and to out-of-memory failures of the controllers at 02:11 on 5 October, and started giving the controllers more memory. Cross-region packet loss began at 03:08. The larger memory allocation started to reach production at 05:27, and intra-region packet loss that started at 07:00 subsided at 08:15 as it rolled out. At 08:22 engineers linked the memory growth to the management plane change, started a rollback at 08:52 and completed it in us-central1 at 09:35, after which no more out-of-memory events happened. A small number of routers still did not accept updates and were restarted by hand without further packet loss. All packet loss had stopped by 10:55, 7 hours and 47 minutes after it started. The Filestore update backlog finished about a day later, on 6 October.

Severity
–
Scope
Networking for newly created and live migrated VMs, cross-region traffic to and from us-central1, intra-region traffic from 07:00 to 08:15, and load balancer health propagation; products that create VMs, including GKE, Cloud SQL, Filestore, Dataproc, Dataflow, Data Fusion, Datastream and Batch
Users affected
–
Region
us-central1
Time to detect
–
Time to resolve
7h 47m

Symptoms

On 5 October 2023 in us-central1, newly created VMs waited a long time before their networking worked, and VMs that had been live migrated lost connectivity for a long time after the migration. Networking telemetry showed cross-region packet loss to and from us-central1 from 03:08 US/Pacific. Health check state of VMs reached load balancers late.

The first signals came hours earlier. On 4 October at 21:45 US/Pacific engineers were alerted to slowness in the virtual network control plane in us-central1, which looked intermittent. At 02:11 on 5 October alerts reported failures in the controllers of the virtual network routers because the controllers ran out of memory. The controllers were restarted automatically but ran out of memory again, and as the out-of-memory events came more often, updates to router state, such as new VMs, new locations of migrated VMs, dynamic routes and health of load balancer backends, fell further behind until practically nothing progressed.

From 03:08 to 10:55 the impact was largely limited to cross-region traffic, and the cross-region routers were hit first and for longer than the intra-region routers. From 07:00 to 08:15 intra-region traffic also lost packets, mainly to and from us-central1-c. Existing VMs that did not migrate or change health were not hit directly, but their traffic could pass through an affected VPN Gateway, internal load balancer or other VM.

Products that create VMs failed with it. 24 percent of GKE cluster creation attempts failed or were delayed, most of them succeeding after up to 120 minutes, and up to 0.4 percent of GKE clusters had downtime or delays during recreate and upgrade operations; in the Google Cloud Console Clusters page some Nodes showed as not registered and the cluster as unhealthy. New Cloud SQL VM creations failed, and Cloud SQL databases became unavailable when a change created a new VM for an existing instance. Filestore upgrades failed and were rolled back, Dataproc cluster creations failed for up to 5 percent, Dataflow jobs could not scale and container downloads from Artifact Registry failed, and Datastream streams showed spikes in the "Data freshness" and "Total latency" metrics.

Symptom Packet lossNetwork unreachableOOMKilledStale endpoint routingCluster or node provisioning stall

Root cause

Root cause Faulty deployment

Component Network control plane (SDN)Cloud load balancer
Contributing factor Excessive blast radiusNo alert on limit
Trigger
A management plane behaviour change, meant to decouple the processing of API updates to Instance Groups and Network Endpoint Groups used as load balancer backends, was deployed in us-central1 after rolling out to several other regions without incident.

Detection

Engineers were alerted to slowness in the virtual network control plane in us-central1 at 21:45 US/Pacific on 4 October, before the packet loss started, and to controller failures from memory exhaustion at 02:11 on 5 October. Networking telemetry showed cross-region packet loss from 03:08.

Remediation

Action items

Lessons

–

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Network control plane (SDN), Cloud load balancer among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.