Cases15 cases

6 to 10 of 15

1. IL-0528 Cloud Pub/Sub could not publish or subscribe in ten regions, and some ordered subscriptions stayed stuck for hours

   Google Cloud2025

   Symptoms

   From 06:54 US/Pacific on 8 January 2025, Google Cloud Pub/Sub customers in ten regions could not publish messages or subscribe to them. Internal telemetry alerted Google engineers at 07:03, nine minutes after the impact started.

   Service outageMonitoring outage

   Trigger

   From 06:54 to 07:30 US/Pacific a bad service configuration change, which over-restricted the permission to access the regional metadata database of Cloud Pub/Sub, was rolled out to multiple regions within a short time.

   Root cause

   Configuration change

   Component

   Message brokerDatabase

   Contributing factor

   Unrepresentative stagingNo staged rolloutIneffective rollback

   Detection

   Internal telemetry alerted Google engineers at 07:03 US/Pacific, nine minutes after the impact started. A second internal telemetry alert at 09:07 showed the ordered subscriptions that could not consume their backlog.

   Remediation

   • Rolled back the configuration change that over-restricted access to the metadata database
   • Identified and repaired every affected ordered subscription so the subscriptions received their backlog

   Action items

   • Enforce parity between pre-production and production environments so configuration changes are caught before production
   • Review change management so configuration changes roll out progressively according to their priority
   • Add monitoring that detects ordering metadata inconsistency
   • Fix the bug in Cloud Pub/Sub ordering metadata management that left ordered messages undelivered

   View case
2. IL-0545 Wasabi API calls in all regions returned HTTP 5XX when the message queue stopped spreading requests over database nodes

   Wasabi2024

   Symptoms

   From 00:17 UTC on 30 November 2024, client connections to Wasabi Cloud Storage and the Web Console failed in every storage region, and all API calls returned HTTP 5XX errors.

   Service outageLoad imbalance

   Root cause

   unknown

   Component

   Message brokerDatabase

   Contributing factor

   –

   Remediation

   • Configured internal servers by hand to route requests across multiple database instances, which restored service
   • After service was restored, worked on the root cause by recovering the internal messaging queue and resuming its automated handling of client requests

   View case
3. IL-0544 Wasabi's database cache stopped responding when its queueing service filled up, and S3 failed in all regions

   Wasabi2024

   Symptoms

   From 10:32 UTC on 6 August 2024, Wasabi's S3 service and user services failed in every region: users had trouble logging into the Web Console and requests to Wasabi regions returned errors.

   Service outageFailover failureIntermittent timeouts

   Trigger

   At 10:32 UTC on 6 August, Wasabi's queueing service reached full capacity, and that made the database cache unresponsive.

   Root cause

   Capacity exhaustion

   Component

   Message brokerIn-memory cacheDatabase

   Contributing factor

   –

   Remediation

   • Restarted the primary database to clear stale connections and cleared the queueing service's queue at the same time, which did not fully restore the database
   • Promoted the secondary database instance to primary, after which S3 was fully operational at 11:20 UTC
   • Restarted the database again between 13:17 and 13:23 UTC to fully incorporate the queueing service library
   • Promoted the original primary database back to production on 7 August, which stopped the user service timeouts
   • Recreated and redeployed the automation cluster that the automation service could not see, which fully restored S3

   View case
4. IL-0285 Clever Cloud's Pulsar brokers stalled in a minor upgrade, then hypervisors overloaded and crashed with kernel panics

   Clever Cloud2024

   Symptoms

   On 2 August 2024, two minutes after Clever Cloud's data team started a minor rolling upgrade of its Apache Pulsar messaging cluster to release 3.3.1, monitoring showed connection problems with the messaging platform. Producing data into the event bus failed, the backlog grew while no new producers or consumers succeeded, and agents on hypervisors, log forwarders on load balancers and deployments were affected. At 10:20 CEST the Pulsar cluster was declared defective.

   Deploys failingSustained high node CPUMemory pressureNode kernel panicOOMKilledAutoscaler misbehaviourMonitoring outage

   Trigger

   At 10:06 CEST the data team began a minor rolling upgrade of Apache Pulsar to release 3.3.1 that also switched on a new balancer for bundle placement: the extensible load manager (loadManagerClassName=org.apache.pulsar.broker.loadbalance.extensions.ExtensibleLoadManagerImpl) with the TransferShedder load shedding strategy instead of the native OverloadShedder.

   Root cause

   Configuration change

   Component

   Message brokerTelemetry agentLinux kernelAutomated capacity manager

   Contributing factor

   No configuration testsMonitoring blind spotIneffective rollbackUnreviewed defaultsAutomation amplificationRetry stormCircular dependency

   Detection

   Monitoring showed connection issues to the messaging platform at 10:08 CEST, two minutes after the upgrade began, and engineers confirmed at 10:09 that producing data into the event bus failed. Monitoring did not catch the first broker's failure to create the system topic, because that broker stayed up.

   Remediation

   • Rolled back the new unload strategy while keeping Pulsar 3.3.1
   • Shut down the client proxies to release the brokers
   • Switched deployment notification events to RabbitMQ as a workaround
   • Stopped all brokers and started only one, which loaded the conflicting system topic so it could be deleted, then started all brokers
   • Restarted the messaging proxies, after which messages flowed again
   • Stopped the watchdog so it would not reload every failed component, and started components one by one
   • Disabled Metrics API ingestion to protect its storage cluster
   • Stopped the long-term storage for observability metrics (the Warp10/FoundationDB cluster) to free resources
   • Put all monitoring instances in all regions into maintenance so they stopped sending events to the orchestrator
   • Killed the overloaded small VMs one by one
   • Flagged 12 hypervisors to drain workload and shut down some load balancers to free CPU and memory
   • Cleaned zombie VMs to speed up recovery
   • Cleared the monitoring alert queue so queued events would not flood the orchestrator
   • Added parked hypervisors to EU-FR-1 to relieve the overloaded ones
   • Removed an overloaded RabbitMQ node from the cluster used by the orchestrator
   • Rolling-restarted every regional L7 load balancer and every L4 load balancer
   • Restarted all HV agents and the add-on provider APIs
   • Destroyed all dangling VMs stuck in the stopping state
   • Re-enabled Metrics API ingestion with a ramp-up QoS policy

   Action items

   • Track KPIs for Pulsar broker and cluster health, hypervisor load, I/O and kernel panic reboots, zombie VMs, RabbitMQ broker and cluster health and deployment queue consumption, each by time of event, detection, decision and resolution
   • Deal with too many monitoring alerts
   • Look at a backup bastion, because a regional bastion is vulnerable
   • Rework the control plane recovery process
   • Rework the correlation between monitoring and orchestration
   • Revisit fault domain separation against multi-tenancy
   • Rethink runtime images and the role of the hypervisor
   • Isolate waved release upgrades of Pulsar
   • Redesign runtime images with better hypervisor integration
   • Regionalise observability for redundancy
   • Set Vector's buffer.max_events and buffer.max_size for small VMs
   • Switch Vector's buffer.type from memory, the default, to disk
   • Change Vector's buffer.when_full from block, the default, to drop_newest
   • Tune journald verbosity and SystemMaxUse on small VMs
   • Add cgroups restrictions for better process control
   • Add I/O restrictions for the VM manager
   • Detect zombie VMs better

   Lessons

   The simulation process that validates non-minor releases was not applied to minor upgrades or to small configuration changes, and this one went straight to production. Running Clever Cloud on Clever Cloud turned into a chicken-and-egg situation: recovery needed the scheduler to carry out large actions while the control plane was itself slowed and the infrastructure had no free resources. Measuring each key indicator by time of event, detection, decision and resolution showed where the team had lacked visibility or reaction and challenged established rules.

   View case
5. IL-0535 An internal Kinesis Data Streams cell in US-EAST-1 degraded, raising errors in CloudWatch Logs, S3 events and Lambda

   Amazon Web Services2024

   Symptoms

   From 2:45 PM PDT on July 30th, 2024, several AWS services in the Northern Virginia (US-EAST-1) Region had higher latency and error rates: CloudWatch Logs, Amazon Data Firehose, the event framework that publishes Amazon S3 events, Amazon ECS, AWS Lambda, Amazon Redshift and AWS Glue. All of them used one internal cell of Amazon Kinesis Data Streams, whose requests had started to see elevated latency and errors.

   High latencyHigh error rateMonitoring outageProbe failures

   Trigger

   At 9:09 AM PDT on July 30th a routine deployment started in one Availability Zone of the Kinesis cell, and taking hosts in and out of service during the deployment set off the failure.

   Root cause

   Software defect

   Component

   Message brokerServerless function platformObject storage

   Contributing factor

   Unrepresentative stagingAutomation amplification

   Remediation

   • Deployed a change that shed load by reducing incoming stream volume from less time-sensitive internal workloads
   • Added capacity to the subsystem that provisions secure connections for Kinesis requests
   • Added more capacity at 6:32 PM PDT to further improve connection provisioning

   Action items

   • Keep the added connection provisioning capacity, load-shedding tooling and changed connection limits in place in US-EAST-1
   • Apply the same changes in every other region that has cells with the new Kinesis Data Streams architecture
   • Update the cell management system to better manage the workload that triggered the issue

   View case

By root cause8