Cases2 cases

1 to 2 of 2

1. IL-0529 New Google Cloud VMs and other instances came up without network connectivity in all zones after VPC controllers stopped

   Google Cloud2024

   Symptoms

   From 15:22 US/Pacific on 16 May 2024, new VMs and other new instances across Google Cloud came up without network programming, so the new instances could not open network connections, and VMs that were migrated or restarted lost connectivity. Changes to virtual network configuration, such as firewall rules and load balancers, were not applied. Internal monitoring alerted Google engineers at 15:41.

   Network unreachablePacket lossCluster or node provisioning stallScale-up failureHigh error rate

   Trigger

   Google's infrastructure team ran a routine maintenance action meant to shut down one unused VPC Controller in a single Google Cloud zone, and the automation shut down VPC controllers in all zones where the controllers were still in use.

   Root cause

   Software defect

   Component

   Network control plane (SDN)Cloud load balancerNAT gateway

   Contributing factor

   Change with a delayed effectIgnored early signalNo safeguard on destructive changeExcessive blast radius

   Detection

   Internal monitoring alerts reached Google engineers at 15:41 US/Pacific, and the investigation started immediately.

   Remediation

   • Paused maintenance operations to prevent the shutdown from repeating
   • Brought the affected VPC Controllers back into a serving state in every zone
   • Stopped the automation responsible for the shutdown until safeguards are in place
   • Recovered partial Apigee proxy deployments by hand
   • Cleaned up VMs left behind by failed Dataproc cluster deletions

   Action items

   • Add safety measures that check that maintenance operations carry correctly scoped zone information
   • Make the underlying tools reject maintenance operations that do not specify zones
   • Tighten on-demand access control so one maintenance invocation cannot work across multiple zones
   • Add safety checks, testing and validation for maintenance operations, including positive and negative test cases for the new safety measures

   View case
2. IL-0532 Azure Software Load Balancer sent packets of one flow to different backends, dropping Firewall and Private Link traffic

   Microsoft Azure2022

   Symptoms

   From 05:26 UTC on 28 June 2022, some Azure customers saw intermittent network failures and packet loss when connecting to resources behind a Software Load Balancer (SLB). Telemetry showed a small increase in errors and customers raised support requests, but the failures were intermittent and were not linked to a recent deployment at first.

   Intermittent connection failuresPacket lossMisrouted requests

   Trigger

   A new SLB code deployment, rolled out through Safe Deployment Practices, made some MUX instances store the backend DIPs of load balancers with more than 15 DIPs in a different order.

   Root cause

   Faulty deployment

   Component

   Cloud load balancer

   Contributing factor

   Ignored early signal

   Detection

   Telemetry showed a small increase in errors and customers raised support requests, but the health checks of Safe Deployment Practices did not correlate these intermittent signals with the deployment until the code had reached 14 regions.

   Remediation

   • Mitigated the VIPs of customers identified as impacted with targeted manual operations
   • Built and tested a hotfix and rolled it out in phases under Safe Deployment Practices
   • Ran automation that scanned all VIPs every five minutes and fixed VIPs in an inconsistent state until the hotfix was out
   • Restored consistent ordering of the backend instances to the state before the incident

   Action items

   • Add Azure Firewall monitoring that detects when the upstream and downstream packets of one flow do not reach the same DIP
   • Keep the IPv6 flow identifier consistent for the lifetime of a connection so all its packets land on one MUX
   • Improve packet capture support for firewalls
   • Add logging of TCP flows, including invalid packets and RST, FIN and FIN-ACK packets
   • Onboard more Azure services to the Gandalf AIOps solution suite for end-to-end monitoring during safe deployments

   Lessons

   Microsoft wrote that the incident showed the need for more thorough end-to-end monitoring during safe deployments.

   View case

By root cause2