Cases4 cases

1 to 4 of 4

1. IL-0366 Cloud Storage uploads and writes in us-central1 slowed and failed with HTTP 503 as traffic piled onto a few backends

   Google Cloud2026

   Symptoms

   From 04:00 US/Pacific on 8 October 2026, Google Cloud Storage upload and write operations in the us-central1 region became slow and failed. Clients mostly saw HTTP 503 UNAVAILABLE errors on WriteObject operations and client-side timeouts.

   High latencyHigh error rateLoad imbalance

   Trigger

   A configuration update to the Cloud Storage load balancing layer turned on a traffic sorting optimization that was meant to deal with local capacity constraints. The recurrence on 9 October came from an incomplete rollback of an upload planner feature.

   Root cause

   Configuration change

   Component

   Object storageDatabase

   Contributing factor

   No headroomOverlapping failures

   Detection

   Google engineering teams monitoring system telemetry detected the problem at 04:50 US/Pacific on 8 October, 50 minutes after it began.

   Remediation

   • Drained traffic from the most overloaded storage backend nodes
   • Deployed a configuration change that disabled the traffic sorting optimization
   • Rolled back a recent update to the metadata database service
   • Redirected traffic and drained affected storage components again during the recurrence
   • Fully reverted the load balancing configuration change, restoring capacity-aware routing
   • Returned the drained nodes to service once resource use was back to normal

   Action items

   • Validate load balancing logic changes in smaller regions before rolling them out to large, highly utilized regions
   • Make the storage load balancing enforce capacity constraints so traffic cannot bypass pre-spilling safeguards
   • Speed up internal data movement within highly utilized storage cells to absorb temporary load spikes

   View case
2. IL-0528 Cloud Pub/Sub could not publish or subscribe in ten regions, and some ordered subscriptions stayed stuck for hours

   Google Cloud2025

   Symptoms

   From 06:54 US/Pacific on 8 January 2025, Google Cloud Pub/Sub customers in ten regions could not publish messages or subscribe to them. Internal telemetry alerted Google engineers at 07:03, nine minutes after the impact started.

   Service outageMonitoring outage

   Trigger

   From 06:54 to 07:30 US/Pacific a bad service configuration change, which over-restricted the permission to access the regional metadata database of Cloud Pub/Sub, was rolled out to multiple regions within a short time.

   Root cause

   Configuration change

   Component

   Message brokerDatabase

   Contributing factor

   Unrepresentative stagingNo staged rolloutIneffective rollback

   Detection

   Internal telemetry alerted Google engineers at 07:03 US/Pacific, nine minutes after the impact started. A second internal telemetry alert at 09:07 showed the ordered subscriptions that could not consume their backlog.

   Remediation

   • Rolled back the configuration change that over-restricted access to the metadata database
   • Identified and repaired every affected ordered subscription so the subscriptions received their backlog

   Action items

   • Enforce parity between pre-production and production environments so configuration changes are caught before production
   • Review change management so configuration changes roll out progressively according to their priority
   • Add monitoring that detects ordering metadata inconsistency
   • Fix the bug in Cloud Pub/Sub ordering metadata management that left ordered messages undelivered

   View case
3. IL-0293 Cloud Bigtable in us-central1 had higher latency and connection errors for about 32 hours from low-rate packet loss

   Google Cloud2023

   Symptoms

   From 19:50 US/Pacific on 4 December 2023, Cloud Bigtable customers in us-central1 saw higher latency and a low level of connection errors, affecting about 0.06% of Bigtable dataplane requests. Automated monitoring first detected the problem at 23:35.

   High latencyHigh error ratePacket lossIntermittent connection failures

   Trigger

   A change that caused an isolation failure on the machines running Cloud Bigtable servers.

   Root cause

   Faulty deployment

   Component

   Database

   Contributing factor

   –

   Detection

   Automated monitoring detected the problem at 23:35 US/Pacific on 4 December 2023.

   Remediation

   • Redirected some traffic away from a subset of the affected machines
   • Rolled back the change that triggered the isolation failure

   Action items

   • Put controls in place to prevent a recurrence

   View case
4. IL-0292 App Engine Memcache was down in all regions for 1 hour 50 minutes when its global datacenter mapping could not be read

   Google Cloud2017

   Symptoms

   From 12:33 US/Pacific on 6 November 2017, the App Engine Memcache service was unavailable to applications in all regions. Google received an automated alert at 12:34, and the status page said all projects using Memcache were affected.

   Service outageDatabase overloadHigh latencyHigh error rate

   Trigger

   A configuration update, after which the database entity holding the mapping of applications to datacenters could no longer be read or written.

   Root cause

   Configuration change

   Component

   In-memory cacheDatabase

   Contributing factor

   Ineffective rollbackExcessive blast radius

   Detection

   Google received an automated alert at 12:34 US/Pacific, one minute after the start.

   Remediation

   • Sent the configuration update with a long enough deadline, which synchronised the database replicas and let clients read the mapping
   • Ramped Memcache traffic back up gradually while the error rate fell

   Action items

   • Reduce the number of readers of the global configuration to avoid contention during writes
   • Regionalise the configuration to limit the blast radius of similar failures

   View case

By root cause2