Cases2 cases

1 to 2 of 2

1. IL-0529 New Google Cloud VMs and other instances came up without network connectivity in all zones after VPC controllers stopped

   Google Cloud2024

   Symptoms

   From 15:22 US/Pacific on 16 May 2024, new VMs and other new instances across Google Cloud came up without network programming, so the new instances could not open network connections, and VMs that were migrated or restarted lost connectivity. Changes to virtual network configuration, such as firewall rules and load balancers, were not applied. Internal monitoring alerted Google engineers at 15:41.

   Network unreachablePacket lossCluster or node provisioning stallScale-up failureHigh error rate

   Trigger

   Google's infrastructure team ran a routine maintenance action meant to shut down one unused VPC Controller in a single Google Cloud zone, and the automation shut down VPC controllers in all zones where the controllers were still in use.

   Root cause

   Software defect

   Component

   Network control plane (SDN)Cloud load balancerNAT gateway

   Contributing factor

   Change with a delayed effectIgnored early signalNo safeguard on destructive changeExcessive blast radius

   Detection

   Internal monitoring alerts reached Google engineers at 15:41 US/Pacific, and the investigation started immediately.

   Remediation

   • Paused maintenance operations to prevent the shutdown from repeating
   • Brought the affected VPC Controllers back into a serving state in every zone
   • Stopped the automation responsible for the shutdown until safeguards are in place
   • Recovered partial Apigee proxy deployments by hand
   • Cleaned up VMs left behind by failed Dataproc cluster deletions

   Action items

   • Add safety measures that check that maintenance operations carry correctly scoped zone information
   • Make the underlying tools reject maintenance operations that do not specify zones
   • Tighten on-demand access control so one maintenance invocation cannot work across multiple zones
   • Add safety checks, testing and validation for maintenance operations, including positive and negative test cases for the new safety measures

   View case
2. IL-0396 New or live-migrated GCE instances, VPN tunnels and network load balancers were unreachable on public IPs for over 2h

   Google Cloud2017

   Symptoms

   On 30 January 2017, Google Compute Engine instances, Cloud VPN tunnels and GCE network load balancers that were created or live migrated between 10:36 and 12:42 PDT could not be reached on their public IP addresses. Resources created earlier, and instances that were not live migrated, kept working.

   Network unreachableProbe failures

   Trigger

   A large set of updates was applied to a rarely used load balancing configuration on the shared layer 2 load balancers.

   Root cause

   Software defect

   Component

   Cloud load balancerNetwork control plane (SDN)

   Contributing factor

   Excessive blast radiusGap in automated tests

   Remediation

   • Restarted the jobs that program changes into the network load balancers, so the stuck changes were processed as a batch
   • Applied the fix zone by zone between 11:36 and 12:42

   Action items

   • Increase the canary timeout so updates that hit the inefficient code path only slow network changes instead of stopping them
   • Improve the inefficient code path
   • Write new tests that check behaviour on a wider range of configurations
   • Accelerate replacing global propagation of address configuration with decentralized routing
   • Add metrics and alerting that identify this kind of issue sooner

   View case

By root cause1