Cases4 cases

1 to 4 of 4

1. IL-0528 Cloud Pub/Sub could not publish or subscribe in ten regions, and some ordered subscriptions stayed stuck for hours

   Google Cloud2025

   Symptoms

   From 06:54 US/Pacific on 8 January 2025, Google Cloud Pub/Sub customers in ten regions could not publish messages or subscribe to them. Internal telemetry alerted Google engineers at 07:03, nine minutes after the impact started.

   Service outageMonitoring outage

   Trigger

   From 06:54 to 07:30 US/Pacific a bad service configuration change, which over-restricted the permission to access the regional metadata database of Cloud Pub/Sub, was rolled out to multiple regions within a short time.

   Root cause

   Configuration change

   Component

   Message brokerDatabase

   Contributing factor

   Unrepresentative stagingNo staged rolloutIneffective rollback

   Detection

   Internal telemetry alerted Google engineers at 07:03 US/Pacific, nine minutes after the impact started. A second internal telemetry alert at 09:07 showed the ordered subscriptions that could not consume their backlog.

   Remediation

   • Rolled back the configuration change that over-restricted access to the metadata database
   • Identified and repaired every affected ordered subscription so the subscriptions received their backlog

   Action items

   • Enforce parity between pre-production and production environments so configuration changes are caught before production
   • Review change management so configuration changes roll out progressively according to their priority
   • Add monitoring that detects ordering metadata inconsistency
   • Fix the bug in Cloud Pub/Sub ordering metadata management that left ordered messages undelivered

   View case
2. IL-0545 Wasabi API calls in all regions returned HTTP 5XX when the message queue stopped spreading requests over database nodes

   Wasabi2024

   Symptoms

   From 00:17 UTC on 30 November 2024, client connections to Wasabi Cloud Storage and the Web Console failed in every storage region, and all API calls returned HTTP 5XX errors.

   Service outageLoad imbalance

   Root cause

   unknown

   Component

   Message brokerDatabase

   Contributing factor

   –

   Remediation

   • Configured internal servers by hand to route requests across multiple database instances, which restored service
   • After service was restored, worked on the root cause by recovering the internal messaging queue and resuming its automated handling of client requests

   View case
3. IL-0544 Wasabi's database cache stopped responding when its queueing service filled up, and S3 failed in all regions

   Wasabi2024

   Symptoms

   From 10:32 UTC on 6 August 2024, Wasabi's S3 service and user services failed in every region: users had trouble logging into the Web Console and requests to Wasabi regions returned errors.

   Service outageFailover failureIntermittent timeouts

   Trigger

   At 10:32 UTC on 6 August, Wasabi's queueing service reached full capacity, and that made the database cache unresponsive.

   Root cause

   Capacity exhaustion

   Component

   Message brokerIn-memory cacheDatabase

   Contributing factor

   –

   Remediation

   • Restarted the primary database to clear stale connections and cleared the queueing service's queue at the same time, which did not fully restore the database
   • Promoted the secondary database instance to primary, after which S3 was fully operational at 11:20 UTC
   • Restarted the database again between 13:17 and 13:23 UTC to fully incorporate the queueing service library
   • Promoted the original primary database back to production on 7 August, which stopped the user service timeouts
   • Recreated and redeployed the automation cluster that the automation service could not see, which fully restored S3

   View case
4. IL-0275 Parse.ly's Kafka brokers hit network limits and failed one after another, stopping the analytics pipeline for hours

   Parse.ly2015

   Symptoms

   On Thursday night, 26 March 2015, and again on Friday, 27 March, Parse.ly's data processing backend stopped. Monitoring tools such as Munin, Sentry and Pingdom reported failures widely, PagerDuty woke the on-call engineer, and no new analytics data appeared in the dashboards at dash.parsely.com and preview.parsely.com for several hours.

   Service outageDNS resolution failures

   Trigger

   Data volume reached a point where the Kafka brokers, running on modest Amazon EC2 nodes without high network capacity, hit operating system network limits under the outbound read traffic and became unavailable.

   Root cause

   Capacity exhaustion

   Component

   Message brokerService discovery

   Contributing factor

   Monitoring blind spotUnknown limitNo headroomHarmful manual action

   Detection

   Munin, Sentry and Pingdom reported the failures widely, Graphite graphs showed them, and PagerDuty woke the on-call engineer when the first outages started in the night.

   Remediation

   • Added new brokers to expand the Kafka cluster
   • Corrected the configuration of the new brokers and deployed them to the cluster, after which network utilization per node dropped and processing resumed
   • Turned the downstream consumer systems back on slowly through Apache Storm and reprocessed the backlog
   • Began restoring the data to customer accounts from the raw data archived in Amazon S3

   Lessons

   CPU and disk I/O are good signs of machine health, but creeping network utilization is harder to monitor, because 100% of network capacity is not well defined on cloud VMs that share host machines. When every node of a cluster runs close to the same limit, losing one node can take down the whole cluster, so failure planning should cover an entire subsystem going offline, not just one or two machines. Keeping raw data archival in simple files and S3 uploads, separate from the Kafka stream, meant there was no risk of permanent data loss.

   View case

By root cause3