Incident Library

IL-0529 Google Cloud · 2024-05-16gcpnetworking

New Google Cloud VMs and other instances came up without network connectivity in all zones after VPC controllers stopped

A separate instance of the VPC Controller runs in every Google Cloud zone and programs the hosts of that zone, such as GCE VM hosts, so that endpoints on them can reach other endpoints in their VPC networks, on-premises destinations through VPN or Interconnect, and the Internet. Within a zone the VPC control plane is split into components that each program a subset of traffic paths, so that a control plane outage stays inside one cluster. On 16 May 2024 maintenance automation was asked to shut down an unused VPC Controller in one zone. Earlier in the year a refactor of the automation software had modified the parameter that specifies the zone, so the target zone was ignored and the shutdown took effect on VPC controllers in all zones. The test environment had caught this failure mode, but the error was misread as an expected failure there, so the change reached production. The automation also had a safeguard that limits the scope of maintenance operations, and that safeguard had been disabled by mistake during a migration to a new version of the automation framework.

The VPC network follows a fail static design, so the data plane kept using the last programmed state and running VMs mostly kept working. Operations that need the control plane broke in all zones: creating new VMs, autoscaling, applying configuration changes to customer projects and reflecting changes in VM health. Because so many products create VMs or change network configuration, the outage spread to GKE, Cloud Run, App Engine Flex, Cloud Functions, Cloud SQL, AlloyDB, Memorystore, Dataflow, Dataproc, Cloud Composer, Cloud Build, Filestore, Cloud NAT, Cloud Interconnect, Cloud VPN, Cloud Router, Cloud Load Balancing and others. The data plane also lost packets during VM live migration and from congestion in some zones.

Internal monitoring alerted engineers at 15:41 US/Pacific. Maintenance operations were paused at once, and in parallel the affected VPC Controllers were brought back into service. Controllers in most zones were working by 17:30, all of them by 18:09, and the backlog of changes to the data plane cleared by 18:40. The main impact lasted 2 hours and 48 minutes. Some products took longer: all remaining Cloud SQL instances were restored by 22:30, some Memorystore for Redis instances recovered at 20:05, some Apigee partial proxy deployments needed manual work until 01:00 on 17 May, several Cloud Composer environments that had started an image upgrade during the disruption were broken beyond recovery, and some NAT configurations took up to two more hours for control plane changes to take effect.

Severity
–
Scope
Network programming for new, migrated and restarted VMs and for network configuration changes in all Google Cloud zones, and every product that creates VMs or changes network configuration, including GKE, Cloud Run, App Engine Flex, Cloud Functions, Cloud SQL, AlloyDB, Memorystore, Dataflow, Dataproc, Cloud Composer, Cloud Build, Filestore, Cloud NAT, Cloud Interconnect, Cloud VPN, Cloud Router and Cloud Load Balancing
Users affected
–
Region
Global (all Google Cloud zones)
Time to detect
19m
Time to resolve
2h 48m

Symptoms

From 15:22 US/Pacific on 16 May 2024, new VMs and other new instances across Google Cloud came up without network programming, so the new instances could not open network connections, and VMs that were migrated or restarted lost connectivity. Changes to virtual network configuration, such as firewall rules and load balancers, were not applied. Internal monitoring alerted Google engineers at 15:41.

The impact reached every product that creates VMs or changes network configuration. All new firewall rules and firewall updates stopped propagating to VM instances in every GCE zone. GKE operations that create or update a VM failed, including cluster and node pool creation and deletion, node upgrades and scaling up an existing node pool. Cloud Run services using Direct VPC Egress could not be deployed or scale up, including from 0, and App Engine Flex deployments could not be created or scale. Batch jobs failed with elevated "NO_VM_REPORT" rates, Cloud SQL create, clone and restore operations failed, newly started Memorystore for Redis VM nodes were unreachable, and Cloud NAT Dynamic Port Allocation (DPA) had allocation failures. Customers of Cloud Interconnect, Cloud VPN, Cloud Router and Cloud NAT could not change those resources, and changes to learned BGP routes did not propagate.

Existing workloads were mostly unaffected because the VPC data plane keeps using its last programmed state when the control plane is unavailable, but there was data plane loss as well. All new endpoints, such as VMs and VPN gateways, and new load balancers saw packet loss to a majority of destinations. 0.07% of VMs migrated during the outage lost connectivity. Long-running VMs in a few locations saw loss from data plane overload, peaking at 3% in us-east1-c and averaging 0.05% across all zones, and in a few GCE zones up to 25% of TCP connections to Google Services were blackholed (us-east4-c), so applications that retried with a different TCP source port saw only limited outages. Some Cloud Load Balancing and Apigee customers got 500 errors, latencies or timeouts.

The status page updates during the incident gave the start as 14:44 US/Pacific and described slow programming in the Cloud Networking control plane and delayed network programming for new and newly migrated VMs. The incident header and the mini incident report give 15:22 as the start.

Symptom Network unreachablePacket lossCluster or node provisioning stallScale-up failureHigh error rate

Root cause

Root cause Software defect

Component Network control plane (SDN)Cloud load balancerNAT gateway
Contributing factor Change with a delayed effectIgnored early signalNo safeguard on destructive changeExcessive blast radius
Trigger
Google's infrastructure team ran a routine maintenance action meant to shut down one unused VPC Controller in a single Google Cloud zone, and the automation shut down VPC controllers in all zones where the controllers were still in use.

Detection

Internal monitoring alerts reached Google engineers at 15:41 US/Pacific, and the investigation started immediately.

Remediation

Action items

Lessons

–

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Network control plane (SDN), Cloud load balancer, NAT gateway among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.