Incident Library
Applied filters Root causeConfiguration changeSymptomHigh error rateOrganisationGoogle CloudClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases3 cases

1 to 3 of 3

  1. IL-0366 Cloud Storage uploads and writes in us-central1 slowed and failed with HTTP 503 as traffic piled onto a few backends
    Google Cloud2026
    Symptoms

    From 04:00 US/Pacific on 8 October 2026, Google Cloud Storage upload and write operations in the us-central1 region became slow and failed. Clients mostly saw HTTP 503 UNAVAILABLE errors on WriteObject operations and client-side timeouts.

    High latencyHigh error rateLoad imbalance
    Trigger

    A configuration update to the Cloud Storage load balancing layer turned on a traffic sorting optimization that was meant to deal with local capacity constraints. The recurrence on 9 October came from an incomplete rollback of an upload planner feature.

    Root cause
    Configuration change
    Component
    Object storageDatabase
    Contributing factor
    No headroomOverlapping failures
    Detection

    Google engineering teams monitoring system telemetry detected the problem at 04:50 US/Pacific on 8 October, 50 minutes after it began.

    Remediation
    • Drained traffic from the most overloaded storage backend nodes
    • Deployed a configuration change that disabled the traffic sorting optimization
    • Rolled back a recent update to the metadata database service
    • Redirected traffic and drained affected storage components again during the recurrence
    • Fully reverted the load balancing configuration change, restoring capacity-aware routing
    • Returned the drained nodes to service once resource use was back to normal
    Action items
    • Validate load balancing logic changes in smaller regions before rolling them out to large, highly utilized regions
    • Make the storage load balancing enforce capacity constraints so traffic cannot bypass pre-spilling safeguards
    • Speed up internal data movement within highly utilized storage cells to absorb temporary load spikes
    View case
  2. IL-0406 Vertex AI Gemini API returned 429 and 503 overload errors for all Gemini models on the global endpoint and in US regions
    Google Cloud2026
    Symptoms

    From 04:37 US/Pacific on 27 February 2026, requests to Vertex AI Gemini API models through the global endpoint and in US regions failed at increased rates with overload errors. PayGo customers mostly got 429 Resource Exhausted, and Provisioned Throughput (PT) customers got 503 Service Unavailable.

    High error rate
    Trigger

    A configuration change to the safety filtering service that supports all Gemini models.

    Root cause
    Configuration change
    Component
    Content safety filter
    Contributing factor
    –
    Detection

    Google's automated monitoring system alerted engineers at 04:54 US/Pacific, and they started investigating at once.

    Remediation
    • Rolled back the faulty configuration change to the previous stable configuration
    • Added more capacity to the safety filtering service to stabilise it
    Action items
    • Reinforce rollout processes with mandatory validation checkpoints
    • Improve alerting to monitor critical dependencies more closely
    View case
  3. IL-0292 App Engine Memcache was down in all regions for 1 hour 50 minutes when its global datacenter mapping could not be read
    Google Cloud2017
    Symptoms

    From 12:33 US/Pacific on 6 November 2017, the App Engine Memcache service was unavailable to applications in all regions. Google received an automated alert at 12:34, and the status page said all projects using Memcache were affected.

    Service outageDatabase overloadHigh latencyHigh error rate
    Trigger

    A configuration update, after which the database entity holding the mapping of applications to datacenters could no longer be read or written.

    Root cause
    Configuration change
    Component
    In-memory cacheDatabase
    Contributing factor
    Ineffective rollbackExcessive blast radius
    Detection

    Google received an automated alert at 12:34 US/Pacific, one minute after the start.

    Remediation
    • Sent the configuration update with a long enough deadline, which synchronised the database replicas and let clients read the mapping
    • Ramped Memcache traffic back up gradually while the error rate fell
    Action items
    • Reduce the number of readers of the global configuration to avoid contention during writes
    • Regionalise the configuration to limit the blast radius of similar failures
    View case
Check your cluster for this

These 3 cases involve Database and In-memory cache. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Configuration change3 cases

Component
Database2In-memory cache1Content safety filter1Object storage1
Contributing factor
Excessive blast radius1No headroom1Overlapping failures1Ineffective rollback1
Organisation
Google Cloud3
  1. IL-0366 Cloud Storage uploads and writes in us-central1 slowed and failed with HTTP 503 as traffic piled onto a few backends
    Google Cloud2026
  2. IL-0406 Vertex AI Gemini API returned 429 and 503 overload errors for all Gemini models on the global endpoint and in US regions
    Google Cloud2026
  3. IL-0292 App Engine Memcache was down in all regions for 1 hour 50 minutes when its global datacenter mapping could not be read
    Google Cloud2017