Cases24 cases

9 to 13 of 24

1. IL-0529 New Google Cloud VMs and other instances came up without network connectivity in all zones after VPC controllers stopped

   Google Cloud2024

   Symptoms

   From 15:22 US/Pacific on 16 May 2024, new VMs and other new instances across Google Cloud came up without network programming, so the new instances could not open network connections, and VMs that were migrated or restarted lost connectivity. Changes to virtual network configuration, such as firewall rules and load balancers, were not applied. Internal monitoring alerted Google engineers at 15:41.

   Network unreachablePacket lossCluster or node provisioning stallScale-up failureHigh error rate

   Trigger

   Google's infrastructure team ran a routine maintenance action meant to shut down one unused VPC Controller in a single Google Cloud zone, and the automation shut down VPC controllers in all zones where the controllers were still in use.

   Root cause

   Software defect

   Component

   Network control plane (SDN)Cloud load balancerNAT gateway

   Contributing factor

   Change with a delayed effectIgnored early signalNo safeguard on destructive changeExcessive blast radius

   Detection

   Internal monitoring alerts reached Google engineers at 15:41 US/Pacific, and the investigation started immediately.

   Remediation

   • Paused maintenance operations to prevent the shutdown from repeating
   • Brought the affected VPC Controllers back into a serving state in every zone
   • Stopped the automation responsible for the shutdown until safeguards are in place
   • Recovered partial Apigee proxy deployments by hand
   • Cleaned up VMs left behind by failed Dataproc cluster deletions

   Action items

   • Add safety measures that check that maintenance operations carry correctly scoped zone information
   • Make the underlying tools reject maintenance operations that do not specify zones
   • Tighten on-demand access control so one maintenance invocation cannot work across multiple zones
   • Add safety checks, testing and validation for maintenance operations, including positive and negative test cases for the new safety measures

   View case
2. IL-0293 Cloud Bigtable in us-central1 had higher latency and connection errors for about 32 hours from low-rate packet loss

   Google Cloud2023

   Symptoms

   From 19:50 US/Pacific on 4 December 2023, Cloud Bigtable customers in us-central1 saw higher latency and a low level of connection errors, affecting about 0.06% of Bigtable dataplane requests. Automated monitoring first detected the problem at 23:35.

   High latencyHigh error ratePacket lossIntermittent connection failures

   Trigger

   A change that caused an isolation failure on the machines running Cloud Bigtable servers.

   Root cause

   Faulty deployment

   Component

   Database

   Contributing factor

   –

   Detection

   Automated monitoring detected the problem at 23:35 US/Pacific on 4 December 2023.

   Remediation

   • Redirected some traffic away from a subset of the affected machines
   • Rolled back the change that triggered the isolation failure

   Action items

   • Put controls in place to prevent a recurrence

   View case
3. IL-0540 Korea's local government administrative services and Government24 went down as a router port dropped most large packets

   Korea Ministry of the Interior and Safety2023

   Symptoms

   On Friday 17 November 2023, the local government administrative computer services in Korea stopped working, and the Government24 service of the National Information Resources Service was down until the next morning.

   Service outagePacket lossHigh latency

   Root cause

   Hardware failure

   Component

   NIC, cable and switch port link

   Contributing factor

   Monitoring blind spotGray failure

   Remediation

   • Replaced the two malfunctioning L4 devices with higher-performance devices, tested them and resumed Government24 at 09:00 on 18 November
   • Moved the connection on the defective port of the Daejeon center router to another port at 07:00 on 19 November
   • Swapped out the module of the router that had the fault

   Action items

   • Inspect all old equipment that may have similar port defects
   • Revise the incident handling manual so the public is told about an outage quickly
   • Announce the status of service recovery quickly through various means
   • Build integrated monitoring of critical service systems and related equipment to catch signs of failure early
   • Form a standing incident response team to send into major outages
   • Prepare administrative measures and a response manual so administrative services can still be provided when core digital government services stop
   • Name a paralysis of the national computer network as a type of disaster and accident, managed from prevention to recovery
   • Build a government-wide crisis response system for linked information systems and strengthen the control tower for outage prevention and response
   • Improve how public IT projects are run and managed, replace hardware and software past their service life quickly and make project pricing realistic
   • Reduce the reliance on outsourcing for developing and operating information systems and bring digital talent into government
   • Review the redundancy, recovery systems and network configuration of the systems in the National Information Resources Service, and its organisation
   • Work with manufacturers on a way to find and monitor failures of internal parts earlier

   Lessons

   The team said that hardware faults like this defective port also happen in the private sector, and that failures of parts inside a device are hard to catch in advance with basic monitoring and daily visual checks. Redundancy did not help because it takes over only when a whole device stops working, and a module that failed only in part did not make the device count as abnormal. The impact was wide because the defective port carried traffic for all the services.

   View case
4. IL-0530 Virtual router controllers in us-central1 ran out of memory in a loop, cutting networking for new and migrated VMs

   Google Cloud2023

   Symptoms

   On 5 October 2023 in us-central1, newly created VMs waited a long time before their networking worked, and VMs that had been live migrated lost connectivity for a long time after the migration. Networking telemetry showed cross-region packet loss to and from us-central1 from 03:08 US/Pacific. Health check state of VMs reached load balancers late.

   Packet lossNetwork unreachableOOMKilledStale endpoint routingCluster or node provisioning stall

   Trigger

   A management plane behaviour change, meant to decouple the processing of API updates to Instance Groups and Network Endpoint Groups used as load balancer backends, was deployed in us-central1 after rolling out to several other regions without incident.

   Root cause

   Faulty deployment

   Component

   Network control plane (SDN)Cloud load balancer

   Contributing factor

   Excessive blast radiusNo alert on limit

   Detection

   Engineers were alerted to slowness in the virtual network control plane in us-central1 at 21:45 US/Pacific on 4 October, before the packet loss started, and to controller failures from memory exhaustion at 02:11 on 5 October. Networking telemetry showed cross-region packet loss from 03:08.

   Remediation

   • Increased the memory allocated to the virtual network router controllers
   • Rolled back the management plane change in us-central1
   • Restarted by hand the routers that were still not accepting updates

   Action items

   • Alert proactively on memory risks and unexpected memory increases
   • Refactor the deployment configuration so engineers can reallocate memory much faster
   • Re-evaluate existing practices and add safety mechanisms for API reconciliation
   • Make management plane changes more visible across teams so the changes can be correlated faster
   • Adjust the deployment footprint so a regional API change is less likely to exhaust memory across the region at once
   • Optimise memory use in the traffic routers and their controllers

   View case
5. IL-0386 After a move to Google Cloud, Postgres connection timeouts and memcached i/o timeouts spiked on GKE nodes

   incident.io2023

   Symptoms

   Shortly after incident.io moved its infrastructure fully into Google Cloud, the application saw a larger-than-normal number of Postgres connection timeouts, and new database connections per second were in the hundreds, spiking up and down. At the same time, the memcached instance used for transient data and faster API responses had sporadic connection timeouts.

   Intermittent timeoutsPacket loss

   Trigger

   After the move to Google Cloud, the application opened hundreds of new Postgres connections per second, and an accidental database join made it send thousands of duplicate calls to an external third party in parallel.

   Root cause

   Capacity exhaustion

   Component

   CNI pluginDatabaseIn-memory cache

   Contributing factor

   Unreviewed defaultsOverlapping failures

   Detection

   An incident was declared when the number of Postgres connection timeouts rose above normal. The remaining cache timeouts were traced through the logs of the Kubernetes node that served an affected request, the request's trace and the node's metrics.

   Remediation

   • Doubled the maximum Postgres connection lifespan from 15m to 30m
   • Made the Postgres connection pools static so they always keep N connections open
   • Raised the memcache client's MaxIdleConns from 2 to 20
   • Moved temporarily from Google's hosted memcached to a memcached inside the Kubernetes cluster
   • Shipped the fix for the accidental database join that made thousands of duplicate external calls

   Action items

   • Set more sensible concurrency limits for outbound network calls

   Lessons

   Network timeouts are not always local to one request or pod: when they appear, the whole activity of the node, including other pods on it, needs to be checked. Staying within a provider's rate limits does not keep outbound calls from affecting a node, so outbound network calls need more sensible concurrency limits.

   View case

By root cause9