Cases2 cases

1 to 2 of 2

1. IL-0528 Cloud Pub/Sub could not publish or subscribe in ten regions, and some ordered subscriptions stayed stuck for hours

   Google Cloud2025

   Symptoms

   From 06:54 US/Pacific on 8 January 2025, Google Cloud Pub/Sub customers in ten regions could not publish messages or subscribe to them. Internal telemetry alerted Google engineers at 07:03, nine minutes after the impact started.

   Service outageMonitoring outage

   Trigger

   From 06:54 to 07:30 US/Pacific a bad service configuration change, which over-restricted the permission to access the regional metadata database of Cloud Pub/Sub, was rolled out to multiple regions within a short time.

   Root cause

   Configuration change

   Component

   Message brokerDatabase

   Contributing factor

   Unrepresentative stagingNo staged rolloutIneffective rollback

   Detection

   Internal telemetry alerted Google engineers at 07:03 US/Pacific, nine minutes after the impact started. A second internal telemetry alert at 09:07 showed the ordered subscriptions that could not consume their backlog.

   Remediation

   • Rolled back the configuration change that over-restricted access to the metadata database
   • Identified and repaired every affected ordered subscription so the subscriptions received their backlog

   Action items

   • Enforce parity between pre-production and production environments so configuration changes are caught before production
   • Review change management so configuration changes roll out progressively according to their priority
   • Add monitoring that detects ordering metadata inconsistency
   • Fix the bug in Cloud Pub/Sub ordering metadata management that left ordered messages undelivered

   View case
2. IL-0375 MongoDB Atlas cluster changes and alerts lagged for hours after a control plane metadata migration congested a database

   MongoDB2024

   Symptoms

   From 18:24 UTC on 3 June 2024, MongoDB Atlas cluster configuration changes in all commercial regions ran with delays of a few minutes to several hours. The delays hit changes requested through the API and changes from automated triggers such as auto-scaling and node replacement.

   Control plane lagMonitoring outageDatabase overload

   Trigger

   During planned internal maintenance, the metadata of an internal workflow management system that backs the Atlas control plane was migrated to a target database cluster, and traffic was redirected to that target.

   Root cause

   Capacity exhaustion

   Component

   Database

   Contributing factor

   Unrepresentative stagingIneffective rollback

   Remediation

   • Rolled back the metadata migration, reconciling data between the source and target databases
   • Executed recovery procedures, completing them at 00:09 UTC on 4 June, and kept monitoring closely

   Action items

   • Design a new metadata migration strategy that can roll back within 15 minutes of detecting a problem
   • Hold similar maintenance activities until the new procedure is implemented and tested

   View case

By root cause2