Graph
Cases4 cases
1 to 4 of 4
IL-0528Cloud Pub/Sub could not publish or subscribe in ten regions, and some ordered subscriptions stayed stuck for hoursGoogle Cloud2025- Symptoms
From 06:54 US/Pacific on 8 January 2025, Google Cloud Pub/Sub customers in ten regions could not publish messages or subscribe to them. Internal telemetry alerted Google engineers at 07:03, nine minutes after the impact started.
- Trigger
From 06:54 to 07:30 US/Pacific a bad service configuration change, which over-restricted the permission to access the regional metadata database of Cloud Pub/Sub, was rolled out to multiple regions within a short time.
- Root cause
- Component
- Contributing factor
- Detection
Internal telemetry alerted Google engineers at 07:03 US/Pacific, nine minutes after the impact started. A second internal telemetry alert at 09:07 showed the ordered subscriptions that could not consume their backlog.
- Remediation
- Rolled back the configuration change that over-restricted access to the metadata database
- Identified and repaired every affected ordered subscription so the subscriptions received their backlog
- Action items
- Enforce parity between pre-production and production environments so configuration changes are caught before production
- Review change management so configuration changes roll out progressively according to their priority
- Add monitoring that detects ordering metadata inconsistency
- Fix the bug in Cloud Pub/Sub ordering metadata management that left ordered messages undelivered
IL-0285Clever Cloud's Pulsar brokers stalled in a minor upgrade, then hypervisors overloaded and crashed with kernel panicsClever Cloud2024- Symptoms
On 2 August 2024, two minutes after Clever Cloud's data team started a minor rolling upgrade of its Apache Pulsar messaging cluster to release 3.3.1, monitoring showed connection problems with the messaging platform. Producing data into the event bus failed, the backlog grew while no new producers or consumers succeeded, and agents on hypervisors, log forwarders on load balancers and deployments were affected. At 10:20 CEST the Pulsar cluster was declared defective.
- Trigger
At 10:06 CEST the data team began a minor rolling upgrade of Apache Pulsar to release 3.3.1 that also switched on a new balancer for bundle placement: the extensible load manager (loadManagerClassName=org.apache.pulsar.broker.loadbalance.extensions.ExtensibleLoadManagerImpl) with the TransferShedder load shedding strategy instead of the native OverloadShedder.
- Root cause
- Component
- Contributing factor
- Detection
Monitoring showed connection issues to the messaging platform at 10:08 CEST, two minutes after the upgrade began, and engineers confirmed at 10:09 that producing data into the event bus failed. Monitoring did not catch the first broker's failure to create the system topic, because that broker stayed up.
- Remediation
- Rolled back the new unload strategy while keeping Pulsar 3.3.1
- Shut down the client proxies to release the brokers
- Switched deployment notification events to RabbitMQ as a workaround
- Stopped all brokers and started only one, which loaded the conflicting system topic so it could be deleted, then started all brokers
- Restarted the messaging proxies, after which messages flowed again
- Stopped the watchdog so it would not reload every failed component, and started components one by one
- Disabled Metrics API ingestion to protect its storage cluster
- Stopped the long-term storage for observability metrics (the Warp10/FoundationDB cluster) to free resources
- Put all monitoring instances in all regions into maintenance so they stopped sending events to the orchestrator
- Killed the overloaded small VMs one by one
- Flagged 12 hypervisors to drain workload and shut down some load balancers to free CPU and memory
- Cleaned zombie VMs to speed up recovery
- Cleared the monitoring alert queue so queued events would not flood the orchestrator
- Added parked hypervisors to EU-FR-1 to relieve the overloaded ones
- Removed an overloaded RabbitMQ node from the cluster used by the orchestrator
- Rolling-restarted every regional L7 load balancer and every L4 load balancer
- Restarted all HV agents and the add-on provider APIs
- Destroyed all dangling VMs stuck in the stopping state
- Re-enabled Metrics API ingestion with a ramp-up QoS policy
- Action items
- Track KPIs for Pulsar broker and cluster health, hypervisor load, I/O and kernel panic reboots, zombie VMs, RabbitMQ broker and cluster health and deployment queue consumption, each by time of event, detection, decision and resolution
- Deal with too many monitoring alerts
- Look at a backup bastion, because a regional bastion is vulnerable
- Rework the control plane recovery process
- Rework the correlation between monitoring and orchestration
- Revisit fault domain separation against multi-tenancy
- Rethink runtime images and the role of the hypervisor
- Isolate waved release upgrades of Pulsar
- Redesign runtime images with better hypervisor integration
- Regionalise observability for redundancy
- Set Vector's buffer.max_events and buffer.max_size for small VMs
- Switch Vector's buffer.type from memory, the default, to disk
- Change Vector's buffer.when_full from block, the default, to drop_newest
- Tune journald verbosity and SystemMaxUse on small VMs
- Add cgroups restrictions for better process control
- Add I/O restrictions for the VM manager
- Detect zombie VMs better
- Lessons
The simulation process that validates non-minor releases was not applied to minor upgrades or to small configuration changes, and this one went straight to production. Running Clever Cloud on Clever Cloud turned into a chicken-and-egg situation: recovery needed the scheduler to carry out large actions while the control plane was itself slowed and the infrastructure had no free resources. Measuring each key indicator by time of event, detection, decision and resolution showed where the team had lacked visibility or reaction and challenged established rules.
IL-0375MongoDB Atlas cluster changes and alerts lagged for hours after a control plane metadata migration congested a databaseMongoDB2024- Symptoms
From 18:24 UTC on 3 June 2024, MongoDB Atlas cluster configuration changes in all commercial regions ran with delays of a few minutes to several hours. The delays hit changes requested through the API and changes from automated triggers such as auto-scaling and node replacement.
- Trigger
During planned internal maintenance, the metadata of an internal workflow management system that backs the Atlas control plane was migrated to a target database cluster, and traffic was redirected to that target.
- Root cause
- Component
- Contributing factor
- Remediation
- Rolled back the metadata migration, reconciling data between the source and target databases
- Executed recovery procedures, completing them at 00:09 UTC on 4 June, and kept monitoring closely
- Action items
- Design a new metadata migration strategy that can roll back within 15 minutes of detecting a problem
- Hold similar maintenance activities until the new procedure is implemented and tested
IL-0222Google Cloud US regions had heavy packet loss and some went unreachable after the network control plane was descheduledGoogle Cloud2019- Symptoms
On Sunday 2 June 2019, with user impact beginning between 11:47 and 11:49 US/Pacific, Google Cloud projects in several US regions saw elevated packet loss from network congestion, and some regions became unreachable. Customers saw higher latency, intermittent errors and lost connectivity to instances in us-central1, us-east1, us-east4, us-west2, northamerica-northeast1 and southamerica-east1, while us-west1, Europe and Asia had no regional congestion.
- Trigger
At 11:45 US/Pacific a relatively rare type of datacenter maintenance event started in one physical location, and the automation that runs it descheduled logical clusters that also held network control plane jobs in other physical locations.
- Root cause
- Component
- Contributing factor
- Detection
Google engineers were alerted to the failure two minutes after it began and started the incident process used for the most serious production incidents. The first customer notice appeared on the status page at 12:53 US/Pacific.
- Remediation
- Lowered the priority of less critical network traffic classes to reduce congestion
- Halted the datacenter automation software responsible for the maintenance event
- Rescheduled the network control plane and its supporting infrastructure in each location
- Rebuilt and redistributed the lost network control plane configuration, rolling it out from 14:03
- Directed service traffic away from the affected regions
- Moved the Cloud VPN control plane in three regions out of its fail-safe state
- Action items
- Keep the maintenance automation off until safeguards stop it descheduling jobs in several physical locations at once
- Harden the cluster management software to reject such requests whatever their origin
- Reconfigure the network control plane to reject maintenance requests of the type involved
- Make the network control plane in each location persist its configuration
- Let the network stay in fail static mode longer when the control plane is lost
- Review, update and test emergency response tooling and procedures, including customer communication tooling, for such network failures
- Extend continuous disaster recovery testing to this and similarly catastrophic failures
These 4 cases involve Database and Message broker. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.
By root cause3
Configuration change2 cases
- Contributing factor
- Organisation
Capacity exhaustion1 case
- Component
- Contributing factor
- Organisation
Software defect1 case
- Contributing factor
- Organisation