Incident Library
Applied filters Root causeConfiguration changeSymptomMonitoring outageOrganisationGoogle CloudClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0528 Cloud Pub/Sub could not publish or subscribe in ten regions, and some ordered subscriptions stayed stuck for hours
    Google Cloud2025
    Symptoms

    From 06:54 US/Pacific on 8 January 2025, Google Cloud Pub/Sub customers in ten regions could not publish messages or subscribe to them. Internal telemetry alerted Google engineers at 07:03, nine minutes after the impact started.

    Service outageMonitoring outage
    Trigger

    From 06:54 to 07:30 US/Pacific a bad service configuration change, which over-restricted the permission to access the regional metadata database of Cloud Pub/Sub, was rolled out to multiple regions within a short time.

    Root cause
    Configuration change
    Component
    Message brokerDatabase
    Contributing factor
    Unrepresentative stagingNo staged rolloutIneffective rollback
    Detection

    Internal telemetry alerted Google engineers at 07:03 US/Pacific, nine minutes after the impact started. A second internal telemetry alert at 09:07 showed the ordered subscriptions that could not consume their backlog.

    Remediation
    • Rolled back the configuration change that over-restricted access to the metadata database
    • Identified and repaired every affected ordered subscription so the subscriptions received their backlog
    Action items
    • Enforce parity between pre-production and production environments so configuration changes are caught before production
    • Review change management so configuration changes roll out progressively according to their priority
    • Add monitoring that detects ordering metadata inconsistency
    • Fix the bug in Cloud Pub/Sub ordering metadata management that left ordered messages undelivered
    View case
Check your cluster for this

This case involves Database and Message broker. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Configuration change1 case

Component
DatabaseMessage broker
Contributing factor
No staged rolloutIneffective rollbackUnrepresentative staging
Organisation
Google Cloud
  1. IL-0528 Cloud Pub/Sub could not publish or subscribe in ten regions, and some ordered subscriptions stayed stuck for hours
    Google Cloud2025