Cases4 cases

1 to 4 of 4

1. IL-0529 New Google Cloud VMs and other instances came up without network connectivity in all zones after VPC controllers stopped

   Google Cloud2024

   Symptoms

   From 15:22 US/Pacific on 16 May 2024, new VMs and other new instances across Google Cloud came up without network programming, so the new instances could not open network connections, and VMs that were migrated or restarted lost connectivity. Changes to virtual network configuration, such as firewall rules and load balancers, were not applied. Internal monitoring alerted Google engineers at 15:41.

   Network unreachablePacket lossCluster or node provisioning stallScale-up failureHigh error rate

   Trigger

   Google's infrastructure team ran a routine maintenance action meant to shut down one unused VPC Controller in a single Google Cloud zone, and the automation shut down VPC controllers in all zones where the controllers were still in use.

   Root cause

   Software defect

   Component

   Network control plane (SDN)Cloud load balancerNAT gateway

   Contributing factor

   Change with a delayed effectIgnored early signalNo safeguard on destructive changeExcessive blast radius

   Detection

   Internal monitoring alerts reached Google engineers at 15:41 US/Pacific, and the investigation started immediately.

   Remediation

   • Paused maintenance operations to prevent the shutdown from repeating
   • Brought the affected VPC Controllers back into a serving state in every zone
   • Stopped the automation responsible for the shutdown until safeguards are in place
   • Recovered partial Apigee proxy deployments by hand
   • Cleaned up VMs left behind by failed Dataproc cluster deletions

   Action items

   • Add safety measures that check that maintenance operations carry correctly scoped zone information
   • Make the underlying tools reject maintenance operations that do not specify zones
   • Tighten on-demand access control so one maintenance invocation cannot work across multiple zones
   • Add safety checks, testing and validation for maintenance operations, including positive and negative test cases for the new safety measures

   View case
2. IL-0293 Cloud Bigtable in us-central1 had higher latency and connection errors for about 32 hours from low-rate packet loss

   Google Cloud2023

   Symptoms

   From 19:50 US/Pacific on 4 December 2023, Cloud Bigtable customers in us-central1 saw higher latency and a low level of connection errors, affecting about 0.06% of Bigtable dataplane requests. Automated monitoring first detected the problem at 23:35.

   High latencyHigh error ratePacket lossIntermittent connection failures

   Trigger

   A change that caused an isolation failure on the machines running Cloud Bigtable servers.

   Root cause

   Faulty deployment

   Component

   Database

   Contributing factor

   –

   Detection

   Automated monitoring detected the problem at 23:35 US/Pacific on 4 December 2023.

   Remediation

   • Redirected some traffic away from a subset of the affected machines
   • Rolled back the change that triggered the isolation failure

   Action items

   • Put controls in place to prevent a recurrence

   View case
3. IL-0530 Virtual router controllers in us-central1 ran out of memory in a loop, cutting networking for new and migrated VMs

   Google Cloud2023

   Symptoms

   On 5 October 2023 in us-central1, newly created VMs waited a long time before their networking worked, and VMs that had been live migrated lost connectivity for a long time after the migration. Networking telemetry showed cross-region packet loss to and from us-central1 from 03:08 US/Pacific. Health check state of VMs reached load balancers late.

   Packet lossNetwork unreachableOOMKilledStale endpoint routingCluster or node provisioning stall

   Trigger

   A management plane behaviour change, meant to decouple the processing of API updates to Instance Groups and Network Endpoint Groups used as load balancer backends, was deployed in us-central1 after rolling out to several other regions without incident.

   Root cause

   Faulty deployment

   Component

   Network control plane (SDN)Cloud load balancer

   Contributing factor

   Excessive blast radiusNo alert on limit

   Detection

   Engineers were alerted to slowness in the virtual network control plane in us-central1 at 21:45 US/Pacific on 4 October, before the packet loss started, and to controller failures from memory exhaustion at 02:11 on 5 October. Networking telemetry showed cross-region packet loss from 03:08.

   Remediation

   • Increased the memory allocated to the virtual network router controllers
   • Rolled back the management plane change in us-central1
   • Restarted by hand the routers that were still not accepting updates

   Action items

   • Alert proactively on memory risks and unexpected memory increases
   • Refactor the deployment configuration so engineers can reallocate memory much faster
   • Re-evaluate existing practices and add safety mechanisms for API reconciliation
   • Make management plane changes more visible across teams so the changes can be correlated faster
   • Adjust the deployment footprint so a regional API change is less likely to exhaust memory across the region at once
   • Optimise memory use in the traffic routers and their controllers

   View case
4. IL-0222 Google Cloud US regions had heavy packet loss and some went unreachable after the network control plane was descheduled

   Google Cloud2019

   Symptoms

   On Sunday 2 June 2019, with user impact beginning between 11:47 and 11:49 US/Pacific, Google Cloud projects in several US regions saw elevated packet loss from network congestion, and some regions became unreachable. Customers saw higher latency, intermittent errors and lost connectivity to instances in us-central1, us-east1, us-east4, us-west2, northamerica-northeast1 and southamerica-east1, while us-west1, Europe and Asia had no regional congestion.

   Packet lossNetwork unreachableHigh latencyHigh error rateMonitoring outage

   Trigger

   At 11:45 US/Pacific a relatively rare type of datacenter maintenance event started in one physical location, and the automation that runs it descheduled logical clusters that also held network control plane jobs in other physical locations.

   Root cause

   Software defect

   Component

   Network control plane (SDN)BGP routingBackbone network (WAN)

   Contributing factor

   Excessive blast radiusCircular dependencyLimited debug accessIneffective rollback

   Detection

   Google engineers were alerted to the failure two minutes after it began and started the incident process used for the most serious production incidents. The first customer notice appeared on the status page at 12:53 US/Pacific.

   Remediation

   • Lowered the priority of less critical network traffic classes to reduce congestion
   • Halted the datacenter automation software responsible for the maintenance event
   • Rescheduled the network control plane and its supporting infrastructure in each location
   • Rebuilt and redistributed the lost network control plane configuration, rolling it out from 14:03
   • Directed service traffic away from the affected regions
   • Moved the Cloud VPN control plane in three regions out of its fail-safe state

   Action items

   • Keep the maintenance automation off until safeguards stop it descheduling jobs in several physical locations at once
   • Harden the cluster management software to reject such requests whatever their origin
   • Reconfigure the network control plane to reject maintenance requests of the type involved
   • Make the network control plane in each location persist its configuration
   • Let the network stay in fail static mode longer when the control plane is lost
   • Review, update and test emergency response tooling and procedures, including customer communication tooling, for such network failures
   • Extend continuous disaster recovery testing to this and similarly catastrophic failures

   View case

By root cause2