Incident Library
  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0528 Cloud Pub/Sub could not publish or subscribe in ten regions, and some ordered subscriptions stayed stuck for hours
    Google Cloud2025
    Symptoms

    From 06:54 US/Pacific on 8 January 2025, Google Cloud Pub/Sub customers in ten regions could not publish messages or subscribe to them. Internal telemetry alerted Google engineers at 07:03, nine minutes after the impact started.

    Service outageMonitoring outage
    Trigger

    From 06:54 to 07:30 US/Pacific a bad service configuration change, which over-restricted the permission to access the regional metadata database of Cloud Pub/Sub, was rolled out to multiple regions within a short time.

    Root cause
    Configuration change
    Component
    Message brokerDatabase
    Contributing factor
    Unrepresentative stagingNo staged rolloutIneffective rollback
    Detection

    Internal telemetry alerted Google engineers at 07:03 US/Pacific, nine minutes after the impact started. A second internal telemetry alert at 09:07 showed the ordered subscriptions that could not consume their backlog.

    Remediation
    • Rolled back the configuration change that over-restricted access to the metadata database
    • Identified and repaired every affected ordered subscription so the subscriptions received their backlog
    Action items
    • Enforce parity between pre-production and production environments so configuration changes are caught before production
    • Review change management so configuration changes roll out progressively according to their priority
    • Add monitoring that detects ordering metadata inconsistency
    • Fix the bug in Cloud Pub/Sub ordering metadata management that left ordered messages undelivered
    View case
  2. IL-0292 App Engine Memcache was down in all regions for 1 hour 50 minutes when its global datacenter mapping could not be read
    Google Cloud2017
    Symptoms

    From 12:33 US/Pacific on 6 November 2017, the App Engine Memcache service was unavailable to applications in all regions. Google received an automated alert at 12:34, and the status page said all projects using Memcache were affected.

    Service outageDatabase overloadHigh latencyHigh error rate
    Trigger

    A configuration update, after which the database entity holding the mapping of applications to datacenters could no longer be read or written.

    Root cause
    Configuration change
    Component
    In-memory cacheDatabase
    Contributing factor
    Ineffective rollbackExcessive blast radius
    Detection

    Google received an automated alert at 12:34 US/Pacific, one minute after the start.

    Remediation
    • Sent the configuration update with a long enough deadline, which synchronised the database replicas and let clients read the mapping
    • Ramped Memcache traffic back up gradually while the error rate fell
    Action items
    • Reduce the number of readers of the global configuration to avoid contention during writes
    • Regionalise the configuration to limit the blast radius of similar failures
    View case
Check your cluster for this

These 2 cases involve Database and In-memory cache. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Configuration change2 cases

Component
Database2In-memory cache1Message broker1
Contributing factor
Ineffective rollback2Excessive blast radius1No staged rollout1Unrepresentative staging1
Organisation
Google Cloud2
  1. IL-0528 Cloud Pub/Sub could not publish or subscribe in ten regions, and some ordered subscriptions stayed stuck for hours
    Google Cloud2025
  2. IL-0292 App Engine Memcache was down in all regions for 1 hour 50 minutes when its global datacenter mapping could not be read
    Google Cloud2017