Cases3 cases

1 to 3 of 3

1. IL-0366 Cloud Storage uploads and writes in us-central1 slowed and failed with HTTP 503 as traffic piled onto a few backends

   Google Cloud2026

   Symptoms

   From 04:00 US/Pacific on 8 October 2026, Google Cloud Storage upload and write operations in the us-central1 region became slow and failed. Clients mostly saw HTTP 503 UNAVAILABLE errors on WriteObject operations and client-side timeouts.

   High latencyHigh error rateLoad imbalance

   Trigger

   A configuration update to the Cloud Storage load balancing layer turned on a traffic sorting optimization that was meant to deal with local capacity constraints. The recurrence on 9 October came from an incomplete rollback of an upload planner feature.

   Root cause

   Configuration change

   Component

   Object storageDatabase

   Contributing factor

   No headroomOverlapping failures

   Detection

   Google engineering teams monitoring system telemetry detected the problem at 04:50 US/Pacific on 8 October, 50 minutes after it began.

   Remediation

   • Drained traffic from the most overloaded storage backend nodes
   • Deployed a configuration change that disabled the traffic sorting optimization
   • Rolled back a recent update to the metadata database service
   • Redirected traffic and drained affected storage components again during the recurrence
   • Fully reverted the load balancing configuration change, restoring capacity-aware routing
   • Returned the drained nodes to service once resource use was back to normal

   Action items

   • Validate load balancing logic changes in smaller regions before rolling them out to large, highly utilized regions
   • Make the storage load balancing enforce capacity constraints so traffic cannot bypass pre-spilling safeguards
   • Speed up internal data movement within highly utilized storage cells to absorb temporary load spikes

   View case
2. IL-0406 Vertex AI Gemini API returned 429 and 503 overload errors for all Gemini models on the global endpoint and in US regions

   Google Cloud2026

   Symptoms

   From 04:37 US/Pacific on 27 February 2026, requests to Vertex AI Gemini API models through the global endpoint and in US regions failed at increased rates with overload errors. PayGo customers mostly got 429 Resource Exhausted, and Provisioned Throughput (PT) customers got 503 Service Unavailable.

   High error rate

   Trigger

   A configuration change to the safety filtering service that supports all Gemini models.

   Root cause

   Configuration change

   Component

   Content safety filter

   Contributing factor

   –

   Detection

   Google's automated monitoring system alerted engineers at 04:54 US/Pacific, and they started investigating at once.

   Remediation

   • Rolled back the faulty configuration change to the previous stable configuration
   • Added more capacity to the safety filtering service to stabilise it

   Action items

   • Reinforce rollout processes with mandatory validation checkpoints
   • Improve alerting to monitor critical dependencies more closely

   View case
3. IL-0292 App Engine Memcache was down in all regions for 1 hour 50 minutes when its global datacenter mapping could not be read

   Google Cloud2017

   Symptoms

   From 12:33 US/Pacific on 6 November 2017, the App Engine Memcache service was unavailable to applications in all regions. Google received an automated alert at 12:34, and the status page said all projects using Memcache were affected.

   Service outageDatabase overloadHigh latencyHigh error rate

   Trigger

   A configuration update, after which the database entity holding the mapping of applications to datacenters could no longer be read or written.

   Root cause

   Configuration change

   Component

   In-memory cacheDatabase

   Contributing factor

   Ineffective rollbackExcessive blast radius

   Detection

   Google received an automated alert at 12:34 US/Pacific, one minute after the start.

   Remediation

   • Sent the configuration update with a long enough deadline, which synchronised the database replicas and let clients read the mapping
   • Ramped Memcache traffic back up gradually while the error rate fell

   Action items

   • Reduce the number of readers of the global configuration to avoid contention during writes
   • Regionalise the configuration to limit the blast radius of similar failures

   View case

By root cause1