Cases2 cases

1 to 2 of 2

1. IL-0529 New Google Cloud VMs and other instances came up without network connectivity in all zones after VPC controllers stopped

   Google Cloud2024

   Symptoms

   From 15:22 US/Pacific on 16 May 2024, new VMs and other new instances across Google Cloud came up without network programming, so the new instances could not open network connections, and VMs that were migrated or restarted lost connectivity. Changes to virtual network configuration, such as firewall rules and load balancers, were not applied. Internal monitoring alerted Google engineers at 15:41.

   Network unreachablePacket lossCluster or node provisioning stallScale-up failureHigh error rate

   Trigger

   Google's infrastructure team ran a routine maintenance action meant to shut down one unused VPC Controller in a single Google Cloud zone, and the automation shut down VPC controllers in all zones where the controllers were still in use.

   Root cause

   Software defect

   Component

   Network control plane (SDN)Cloud load balancerNAT gateway

   Contributing factor

   Change with a delayed effectIgnored early signalNo safeguard on destructive changeExcessive blast radius

   Detection

   Internal monitoring alerts reached Google engineers at 15:41 US/Pacific, and the investigation started immediately.

   Remediation

   • Paused maintenance operations to prevent the shutdown from repeating
   • Brought the affected VPC Controllers back into a serving state in every zone
   • Stopped the automation responsible for the shutdown until safeguards are in place
   • Recovered partial Apigee proxy deployments by hand
   • Cleaned up VMs left behind by failed Dataproc cluster deletions

   Action items

   • Add safety measures that check that maintenance operations carry correctly scoped zone information
   • Make the underlying tools reject maintenance operations that do not specify zones
   • Tighten on-demand access control so one maintenance invocation cannot work across multiple zones
   • Add safety checks, testing and validation for maintenance operations, including positive and negative test cases for the new safety measures

   View case
2. IL-0336 AWS Direct Connect traffic into the Tokyo Region had packet loss and intermittent connectivity for about six hours

   Amazon Web Services2021

   Symptoms

   From 7:30 AM JST on 2 September 2021, AWS Direct Connect customers saw intermittent connectivity problems and elevated packet loss on traffic going to their VPCs in the Tokyo (AP-NORTHEAST-1) Region. Internal AWS alarms on Direct Connect packet loss fired at the same time.

   Packet lossIntermittent connection failures

   Trigger

   Customer traffic consistently generated packets with a very specific set of attributes and contents, and these packets set off a latent defect in the network device operating system on one layer of the Direct Connect network in Tokyo.

   Root cause

   Software defect

   Component

   Transit gateway or network interconnect

   Contributing factor

   Change with a delayed effectUnrepresentative staging

   Detection

   Internal alarms alerted AWS engineers to elevated packet loss for Direct Connect customers connecting to the Tokyo Region at 7:30 AM JST. The automation that monitors network devices also saw a higher rate of failed devices than normal and alerted engineers.

   Remediation

   • Removed the failing network devices from service so healthy devices carried the traffic, which helped only for a time
   • Reset failed devices and brought them back into service slowly, without stopping the failures
   • Disabled the new protocol in a single Availability Zone, then across the Tokyo Region

   Action items

   • Apply the change carefully to all other AWS Regions, with a new way to detect and remediate the issue before customers are affected

   View case

By root cause1