IL-0396 Google Cloud · 2017-01-30gcpnetworking

New or live-migrated GCE instances, VPN tunnels and network load balancers were unreachable on public IPs for over 2h

All inbound traffic to GCE instances, load balancers and VPN tunnels enters through shared layer 2 load balancers. Changes to the IP addresses of these resources are configured on those load balancers, tested automatically in a canary deployment and then propagated globally. On 30 January 2017 a large set of updates to a rarely used load balancing configuration hit an inefficient code path, and the canary timed out. Every later change of public addressing queued behind those changes, which could not get past the testing phase, so no new or live-migrated resource got working public addressing for 2 hours 8 minutes.

Google engineers restarted the jobs that program changes into the network load balancers. After the restart the problem changes were processed as a batch, which did not reach the inefficient code path, and queued updates went through again. The fix was applied zone by zone between 11:36 and 12:42.

Severity: –
Scope: GCE instances, Cloud VPN tunnels and GCE network load balancers created or live migrated between 10:36 and 12:42 PDT were unreachable on public IPs and could not send outbound traffic; existing resources were unaffected
Users affected: –
Region: all GCE regions; europe-west1 was the last to clear
Time to detect: –
Time to resolve: 2h 8m

Symptoms

On 30 January 2017, Google Compute Engine instances, Cloud VPN tunnels and GCE network load balancers that were created or live migrated between 10:36 and 12:42 PDT could not be reached on their public IP addresses. Resources created earlier, and instances that were not live migrated, kept working.

The affected instances also could not send outbound traffic, and load balancing health checks against them did not succeed. Google's status page first described the problem at 11:13 as newly created VMs having no network connectivity, also affecting VMs undergoing live migration, and at 12:40 added that newly created VPNs were affected as well.

Recovery went zone by zone. At 12:07 most GCE regions were reported mitigated, at 12:40 all regions except europe-west1, which was still clearing. Connectivity for new GCE instances was fully back at 12:45 and for VPNs at 12:50.

Symptom Network unreachable Probe failures

Root cause

Root cause Software defect

Component Cloud load balancer Network control plane (SDN)

Contributing factor Excessive blast radius Gap in automated tests

Trigger: A large set of updates was applied to a rarely used load balancing configuration on the shared layer 2 load balancers.

Detection

–

Remediation

Restarted the jobs that program changes into the network load balancers, so the stuck changes were processed as a batch
Applied the fix zone by zone between 11:36 and 12:42

Action items

Increase the canary timeout so updates that hit the inefficient code path only slow network changes instead of stopping them
Improve the inefficient code path
Write new tests that check behaviour on a wider range of configurations
Accelerate replacing global propagation of address configuration with decentralized routing
Add metrics and alerting that identify this kind of issue sooner

Lessons

–

Sources

https://status.cloud.google.com/incident/compute/17003 · official-status · archived copy

Related cases

IL-0529 New Google Cloud VMs and other instances came up without network connectivity in all zones after VPC controllers stoppedGoogle Cloud · 2024
same root cause · shared Network control plane (SDN), Cloud load balancer
IL-0150 A latent race condition in DynamoDB's DNS automation left an empty DNS record for dynamodb.us-east-1.amazonaws.com, and the DynamoDB and EC2 launch failures that followed caused EKS, ECS and Fargate container launch failures and cluster scaling delays in us-east-1Amazon Web Services · 2025
same root cause · shared Cloud load balancer
IL-0352 Heroku Router 2.0 spread HTTP requests unevenly across dynos for four days, causing latency and timeoutsHeroku · 2025
same root cause · shared Cloud load balancer
IL-0233 A VPC control plane release broke IP alias routes in Nebius eu-west1, and Managed Kubernetes pods lost connectivityNebius · 2025
same root cause · shared Network control plane (SDN)
IL-0347 About 25% of Scaleway FR-PAR-1 hypervisors lost VPC connectivity when both VPC BGP route reflectors got stuckScaleway · 2024
same root cause · shared Network control plane (SDN)
IL-0131 A new kind of GenerateContent request made Vertex AI prediction API servers segfault, and once enough of them restarted at the same time the serving fleet could not carry the loadGoogle Cloud · 2024
same root cause · shared Cloud load balancer
IL-0253 Firefox froze and could not load any web content for close to two hours, stuck in an HTTP/3 request to TelemetryMozilla · 2022
same root cause · shared Cloud load balancer
IL-0243 Google's global External Proxy Load Balancing returned 404 errors for most HTTP/S endpoints for 33 minutesGoogle Cloud · 2021
same root cause · shared Cloud load balancer
IL-0106 A malformed Traefik IngressRoute triggered a Traefik v2 bug once a node rollover restarted every Traefik pod, and all Kubernetes services returned 404DFDS · 2021
same root cause · shared Cloud load balancer
IL-0493 Netlify's Google Cloud TCP load balancer stopped serving about 5% of global traffic as all backends failed health checksNetlify · 2021
same root cause · shared Cloud load balancer
IL-0345 Many Render services in the Oregon region returned 503 errors for about four hoursRender · 2020
same root cause · shared Cloud load balancer
IL-0222 Google Cloud US regions had heavy packet loss and some went unreachable after the network control plane was descheduledGoogle Cloud · 2019
same root cause · shared Network control plane (SDN)
IL-0030 Most nodes of a cluster went NotReady together for 15 minutes while the API server sat behind an AWS ELBZalando · 2019
same root cause · shared Cloud load balancer
IL-0338 Google Cloud HTTP(S) and TCP/SSL Proxy load balancers returned 502s or reset connections on 33% to 87% of requestsGoogle Cloud · 2018
same root cause · shared Cloud load balancer
IL-0395 Google Cloud HTTP(S) Load Balancer gave 502s on 25% of requests for 22 minutes after reverting to a stale configurationGoogle Cloud · 2017
same root cause · shared Cloud load balancer

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Cloud load balancer, Network control plane (SDN) among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0396
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0396/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.