Cases3 cases

1 to 3 of 3

1. IL-0529 New Google Cloud VMs and other instances came up without network connectivity in all zones after VPC controllers stopped

   Google Cloud2024

   Symptoms

   From 15:22 US/Pacific on 16 May 2024, new VMs and other new instances across Google Cloud came up without network programming, so the new instances could not open network connections, and VMs that were migrated or restarted lost connectivity. Changes to virtual network configuration, such as firewall rules and load balancers, were not applied. Internal monitoring alerted Google engineers at 15:41.

   Network unreachablePacket lossCluster or node provisioning stallScale-up failureHigh error rate

   Trigger

   Google's infrastructure team ran a routine maintenance action meant to shut down one unused VPC Controller in a single Google Cloud zone, and the automation shut down VPC controllers in all zones where the controllers were still in use.

   Root cause

   Software defect

   Component

   Network control plane (SDN)Cloud load balancerNAT gateway

   Contributing factor

   Change with a delayed effectIgnored early signalNo safeguard on destructive changeExcessive blast radius

   Detection

   Internal monitoring alerts reached Google engineers at 15:41 US/Pacific, and the investigation started immediately.

   Remediation

   • Paused maintenance operations to prevent the shutdown from repeating
   • Brought the affected VPC Controllers back into a serving state in every zone
   • Stopped the automation responsible for the shutdown until safeguards are in place
   • Recovered partial Apigee proxy deployments by hand
   • Cleaned up VMs left behind by failed Dataproc cluster deletions

   Action items

   • Add safety measures that check that maintenance operations carry correctly scoped zone information
   • Make the underlying tools reject maintenance operations that do not specify zones
   • Tighten on-demand access control so one maintenance invocation cannot work across multiple zones
   • Add safety checks, testing and validation for maintenance operations, including positive and negative test cases for the new safety measures

   View case
2. IL-0267 Wikimedia's eqsin cache site went unreachable when both of its transport links to the core data centers were down

   Wikimedia Foundation2023

   Symptoms

   On 10 January 2023 at 16:37 UTC Icinga reported "BGP status on cr3-eqsin is CRITICAL: BGP CRITICAL - AS1299/IPv6: Active - Telia, AS1299/IPv4: Active - Telia", and from 16:45 the hosts in Wikimedia's eqsin site went DOWN with "PING CRITICAL - Packet loss = 100%". Users normally directed to eqsin, mostly in the APAC region, could only read Wikipedia pages already cached in eqsin.

   Network unreachable

   Trigger

   Planned maintenance by the transport provider took down eqsin's second transport link while the first was still down from a long fiber cut.

   Root cause

   Dependency failure

   Component

   Backbone network (WAN)BGP routingCDN and edge network

   Contributing factor

   Single point of failureIgnored early signal

   Detection

   Automated monitoring: Icinga BGP status alerts for cr3-eqsin at 16:37 and cr2-eqsin at 16:44, followed by a flood of host-down alerts and three pages at the same time.

   Remediation

   • Depooled eqsin from DNS
   • Repooled eqsin

   Action items

   • Create a backup GRE tunnel for eqsin (T327265)
   • Ensure the long-down transport link comes back up properly once it is fixed (T322529)
   • Automatically parse maintenance notifications and alert on conflicting maintenance (T230835)

   Lessons

   The site was depooled quickly, and "depool first, investigate later" was the right attitude. A flood of host-down alerts usually means a network problem. eqsin stayed too long on a single working transport link, SREs did not catch that the downtime of the two transport links overlapped, and the provider's planned maintenance was not recorded in a task or reminder.

   View case
3. IL-0245 Acquia Cloud Enterprise sites in US-East went down when their web servers could no longer reach the Memcache servers

   Acquia2019

   Symptoms

   On 7 February 2019 from 22:17 to 23:25 UTC, some Acquia Cloud Enterprise sites in US-East became unstable or went down for multiple customers. The web servers could not reach the Memcache servers, PHP processes on the web servers were blocked, and the sites became unavailable.

   Service outageNetwork unreachable

   Trigger

   While setting up a temporary file system to store diagnostic data, a rule was deleted by mistake from a VPC security group.

   Root cause

   Operator error

   Component

   VPC route tables and firewall rulesIn-memory cache

   Contributing factor

   Ignored early signal

   Detection

   Acquia's configuration drift alerting fired, but the alert was not seen in time to shorten the outage.

   Remediation

   • Restored the missing rule in the security group by hand

   Action items

   • Correct the internal workflow with all involved teams to prevent similar mistakes
   • Alert internal teams faster about configuration drift in VPC security groups

   View case

By root cause3