Incident Library

IL-0528 Google Cloud · 2025-01-08gcpdatabase

Cloud Pub/Sub could not publish or subscribe in ten regions, and some ordered subscriptions stayed stuck for hours

Cloud Pub/Sub keeps the metadata of its storage system in a regional database, including which messages were published and in what order the messages were published for ordered delivery. Most data plane operations of Cloud Pub/Sub need this database. On 8 January 2025, from 06:54 to 07:30 US/Pacific, a service configuration change that unintentionally over-restricted the permission to access this database was rolled out to multiple regions. The problem did not show in the pre-production environment because its configuration did not match production, and the change went to several regions within a short time instead of following the standard rollout process. Cloud Pub/Sub could no longer read the metadata store, so publish, subscribe and backlog metrics failed in ten regions. Cloud Logging exports and BigQuery Data Transfer Service runs that go through Pub/Sub were also affected. Engineers identified the change and completed its rollback at 08:07, 1 hour and 13 minutes after the start.

The long database unavailability exposed a latent bug in how Cloud Pub/Sub enforces ordered delivery. For subscriptions with ordering enabled, the ordering metadata became inconsistent with the metadata about published messages, and that inconsistency kept a subset of messages from being delivered after the rollback. Telemetry flagged the stuck ordered subscriptions at 09:07, the cause was found at 12:20, and engineers identified and repaired every affected ordered subscription by 23:09, after which the subscriptions received their whole backlog in the proper order.

Severity
–
Scope
Cloud Pub/Sub publish and subscribe, and backlog metrics, in ten regions for 1 hour and 13 minutes; a small subset of ordered subscriptions could not consume their backlog until 23:09; Cloud Logging exports to Pub/Sub delayed for 26 minutes; BigQuery Data Transfer Service runs failed for 20 minutes
Users affected
–
Region
europe-west10, asia-south1, europe-west1, us-central1, asia-southeast2, us-east1, us-east5, asia-south2, us-south1, me-central1
Time to detect
9m
Time to resolve
16h 15m

Symptoms

From 06:54 US/Pacific on 8 January 2025, Google Cloud Pub/Sub customers in ten regions could not publish messages or subscribe to them. Internal telemetry alerted Google engineers at 07:03, nine minutes after the impact started.

The affected regions were europe-west10, asia-south1, europe-west1, us-central1, asia-southeast2, us-east1, us-east5, asia-south2, us-south1 and me-central1. Publishing from other regions could fail as well when the message storage policy stored and processed messages in one of these regions. Pub/Sub backlog metrics could be stale or missing. Cloud Logging exports to Pub/Sub were delayed for 26 minutes, and BigQuery Data Transfer Service runs failed to publish to Pub/Sub for 20 minutes.

Publish and subscribe recovered at 08:07, but at 09:07 internal telemetry showed that a small subset of subscriptions with message ordering enabled could not consume their backlog. Their backlog kept growing after the main outage had ended. Google found the cause, inconsistent ordering metadata, at 12:20, and the last of these subscriptions was repaired at 23:09. Message ordering and guaranteed delivery were not broken; the held-back messages were delivered in order once each subscription was repaired.

Symptom Service outageMonitoring outage

Root cause

Root cause Configuration change

Component Message brokerDatabase
Contributing factor Unrepresentative stagingNo staged rolloutIneffective rollback
Trigger
From 06:54 to 07:30 US/Pacific a bad service configuration change, which over-restricted the permission to access the regional metadata database of Cloud Pub/Sub, was rolled out to multiple regions within a short time.

Detection

Internal telemetry alerted Google engineers at 07:03 US/Pacific, nine minutes after the impact started. A second internal telemetry alert at 09:07 showed the ordered subscriptions that could not consume their backlog.

Remediation

Action items

Lessons

–

Sources

References

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Message broker, Database among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.