Cases5 cases
1 to 5 of 5
IL-0285Clever Cloud's Pulsar brokers stalled in a minor upgrade, then hypervisors overloaded and crashed with kernel panicsClever Cloud2024- Symptoms
On 2 August 2024, two minutes after Clever Cloud's data team started a minor rolling upgrade of its Apache Pulsar messaging cluster to release 3.3.1, monitoring showed connection problems with the messaging platform. Producing data into the event bus failed, the backlog grew while no new producers or consumers succeeded, and agents on hypervisors, log forwarders on load balancers and deployments were affected. At 10:20 CEST the Pulsar cluster was declared defective.
- Trigger
At 10:06 CEST the data team began a minor rolling upgrade of Apache Pulsar to release 3.3.1 that also switched on a new balancer for bundle placement: the extensible load manager (loadManagerClassName=org.apache.pulsar.broker.loadbalance.extensions.ExtensibleLoadManagerImpl) with the TransferShedder load shedding strategy instead of the native OverloadShedder.
- Root cause
- Component
- Contributing factor
- Detection
Monitoring showed connection issues to the messaging platform at 10:08 CEST, two minutes after the upgrade began, and engineers confirmed at 10:09 that producing data into the event bus failed. Monitoring did not catch the first broker's failure to create the system topic, because that broker stayed up.
- Remediation
- Rolled back the new unload strategy while keeping Pulsar 3.3.1
- Shut down the client proxies to release the brokers
- Switched deployment notification events to RabbitMQ as a workaround
- Stopped all brokers and started only one, which loaded the conflicting system topic so it could be deleted, then started all brokers
- Restarted the messaging proxies, after which messages flowed again
- Stopped the watchdog so it would not reload every failed component, and started components one by one
- Disabled Metrics API ingestion to protect its storage cluster
- Stopped the long-term storage for observability metrics (the Warp10/FoundationDB cluster) to free resources
- Put all monitoring instances in all regions into maintenance so they stopped sending events to the orchestrator
- Killed the overloaded small VMs one by one
- Flagged 12 hypervisors to drain workload and shut down some load balancers to free CPU and memory
- Cleaned zombie VMs to speed up recovery
- Cleared the monitoring alert queue so queued events would not flood the orchestrator
- Added parked hypervisors to EU-FR-1 to relieve the overloaded ones
- Removed an overloaded RabbitMQ node from the cluster used by the orchestrator
- Rolling-restarted every regional L7 load balancer and every L4 load balancer
- Restarted all HV agents and the add-on provider APIs
- Destroyed all dangling VMs stuck in the stopping state
- Re-enabled Metrics API ingestion with a ramp-up QoS policy
- Action items
- Track KPIs for Pulsar broker and cluster health, hypervisor load, I/O and kernel panic reboots, zombie VMs, RabbitMQ broker and cluster health and deployment queue consumption, each by time of event, detection, decision and resolution
- Deal with too many monitoring alerts
- Look at a backup bastion, because a regional bastion is vulnerable
- Rework the control plane recovery process
- Rework the correlation between monitoring and orchestration
- Revisit fault domain separation against multi-tenancy
- Rethink runtime images and the role of the hypervisor
- Isolate waved release upgrades of Pulsar
- Redesign runtime images with better hypervisor integration
- Regionalise observability for redundancy
- Set Vector's buffer.max_events and buffer.max_size for small VMs
- Switch Vector's buffer.type from memory, the default, to disk
- Change Vector's buffer.when_full from block, the default, to drop_newest
- Tune journald verbosity and SystemMaxUse on small VMs
- Add cgroups restrictions for better process control
- Add I/O restrictions for the VM manager
- Detect zombie VMs better
- Lessons
The simulation process that validates non-minor releases was not applied to minor upgrades or to small configuration changes, and this one went straight to production. Running Clever Cloud on Clever Cloud turned into a chicken-and-egg situation: recovery needed the scheduler to carry out large actions while the control plane was itself slowed and the infrastructure had no free resources. Measuring each key indicator by time of event, detection, decision and resolution showed where the team had lacked visibility or reaction and challenged established rules.
- Symptoms
From 7:30 AM PST on 7 December 2021, AWS services in us-east-1 that communicate between AWS's internal network and its main network saw rising latency and errors. The AWS operations teams lost real-time monitoring data at once, and the first thing found in the logs was elevated internal DNS errors.
- Trigger
At 7:30 AM PST an automated activity that scales the capacity of one AWS service hosted in the main AWS network triggered unexpected behaviour from a large number of clients inside the internal network.
- Root cause
- Component
- Contributing factor
- Detection
The congestion immediately cut off real-time monitoring data for the AWS operations teams, who turned to logs and first identified elevated internal DNS errors.
- Remediation
- Moved internal DNS traffic away from the congested network paths
- Isolated the top sources of traffic onto dedicated network devices
- Disabled some heavy network traffic services, including EventBridge event delivery
- Brought additional networking capacity online
- Disabled the automated scaling activities that triggered the event
- Action items
- Keep the triggering scaling activities off until all remediations are deployed
- Fix the latent issue that kept networking clients from backing off
- Deploy network configuration that protects the affected networking devices in a similar congestion event
- Keep API Gateway servers healthy when the internal network is unavailable and speed up their recycling
- Release a new Service Health Dashboard that makes service impact easier to understand
- Run the support system actively across multiple AWS regions
IL-0220An overloaded AWS Transit Gateway dropped packets on 4 January 2021, and Slack's web tier saturated and went downSlack2021- Symptoms
During the Americas' morning of 4 January 2021, an external monitoring service paged Slack because error rates were creeping up. While the investigation was starting, Slack's own dashboarding and alerting service became unavailable, so responders had to work without their usual dashboards and alerts.
- Trigger
On the first working day after the holidays, Slack's traffic went from the quietest time of the year to one of its biggest days overnight, with cold client caches pulling more data than usual, and one of Slack's AWS Transit Gateways did not scale fast enough for it.
- Root cause
- Component
- Contributing factor
- Detection
An external monitoring service paged Slack when error rates started creeping up during the Americas' morning. AWS engineers were separately alerted to the packet drops by AWS's internal monitoring.
- Remediation
- Disabled automated downscaling of the web tier to keep serving capacity and allow investigation
- Cleared up the broken, partly provisioned instances that filled the autoscaling groups
- Repaired provision-service so new web tier instances could be provisioned and enter service
- AWS engineers increased the Transit Gateway capacity manually across all Availability Zones
- Action items
- AWS reviews its Transit Gateway scaling algorithms for large packet-per-second increases
- Request a preemptive upscaling of the Transit Gateways at the end of the next holiday season
- Run the dashboard service instances in the same VPC as their database to remove the Transit Gateway dependency
- Load test provision-service regularly
- Reevaluate health-checking and autoscaling configurations so they cannot overload provision-service again
- Lessons
Monitoring has to stay as independent as possible of the infrastructure it watches, or it fails when it is needed most. Here the dashboard and alerting services failed because they ran in a different VPC from their databases and so depended on the Transit Gateways. provision-service had been load tested before, but this event exceeded the earlier scale-ups.
- Symptoms
At 5:15 AM PST on 25 November 2020, alarms fired for errors on putting and getting Amazon Kinesis records in the Northern Virginia (US-EAST-1) Region. From the same time CloudWatch, which processes metric and log data through Kinesis Data Streams, and CloudWatch Events and EventBridge returned more errors.
- Trigger
A relatively small addition of capacity to the Kinesis front-end fleet, made between 2:44 AM and 3:47 AM PST.
- Root cause
- Component
- Contributing factor
- Detection
Alarms for errors on putting and getting Kinesis records fired at 5:15 AM PST, and the teams began reviewing logs.
- Remediation
- Removed the newly added front-end capacity
- Added a configuration so front-end servers load bootstrap data from the authoritative metadata store instead of neighbouring servers
- Restarted the front-end fleet gradually, a few hundred servers per hour from 10:07 AM PST
- Added Cognito capacity to buffer more calls to Kinesis
- Deployed a change that reduced Cognito's dependency on Kinesis
- Mitigated the memory contention on the Lambda invocation hosts
- Informed customers through the backup Service Health Dashboard tool, the Personal Health Dashboard and a global banner
- Action items
- Move the Kinesis front-end to larger CPU and memory servers so fewer servers, and fewer threads per server, are needed
- Add fine-grained alarming for thread consumption
- Finish testing a higher operating system thread count limit
- Greatly improve the cold-start time of the front-end fleet
- Move the front-end server cache to a dedicated fleet
- Move a few large AWS services such as CloudWatch to a separate, partitioned front-end fleet
- Accelerate the cellularization of the front-end fleet to match the back-end
- Change the Cognito webservers so they sustain Kinesis API errors without exhausting their buffers
- Keep 3 hours of metric data in the CloudWatch local metrics data store, first in US-EAST-1 and then globally
- Train support engineers regularly on the backup tool for posting to the Service Health Dashboard
- Lessons
In the Kinesis front-end, the number of threads each server keeps grows directly with the number of servers in the fleet, so adding servers can push every server past an operating system limit at the same time. AWS sees cellularization of the front-end, already done for the back-end, as the way to keep the shard-map cache within a tested and operated range and to guard against unknown scaling limits.
IL-0222Google Cloud US regions had heavy packet loss and some went unreachable after the network control plane was descheduledGoogle Cloud2019- Symptoms
On Sunday 2 June 2019, with user impact beginning between 11:47 and 11:49 US/Pacific, Google Cloud projects in several US regions saw elevated packet loss from network congestion, and some regions became unreachable. Customers saw higher latency, intermittent errors and lost connectivity to instances in us-central1, us-east1, us-east4, us-west2, northamerica-northeast1 and southamerica-east1, while us-west1, Europe and Asia had no regional congestion.
- Trigger
At 11:45 US/Pacific a relatively rare type of datacenter maintenance event started in one physical location, and the automation that runs it descheduled logical clusters that also held network control plane jobs in other physical locations.
- Root cause
- Component
- Contributing factor
- Detection
Google engineers were alerted to the failure two minutes after it began and started the incident process used for the most serious production incidents. The first customer notice appeared on the status page at 12:53 US/Pacific.
- Remediation
- Lowered the priority of less critical network traffic classes to reduce congestion
- Halted the datacenter automation software responsible for the maintenance event
- Rescheduled the network control plane and its supporting infrastructure in each location
- Rebuilt and redistributed the lost network control plane configuration, rolling it out from 14:03
- Directed service traffic away from the affected regions
- Moved the Cloud VPN control plane in three regions out of its fail-safe state
- Action items
- Keep the maintenance automation off until safeguards stop it descheduling jobs in several physical locations at once
- Harden the cluster management software to reject such requests whatever their origin
- Reconfigure the network control plane to reject maintenance requests of the type involved
- Make the network control plane in each location persist its configuration
- Let the network stay in fail static mode longer when the control plane is lost
- Review, update and test emergency response tooling and procedures, including customer communication tooling, for such network failures
- Extend continuous disaster recovery testing to this and similarly catastrophic failures
These 5 cases involve Automated capacity manager and Transit gateway or network interconnect. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.
By root cause4
Configuration change1 case
- Contributing factor
- Organisation
Software defect2 cases
- Component
- Contributing factor
- Organisation
Cloud provider outage1 case
- Contributing factor
- Organisation
Capacity exhaustion1 case
- Contributing factor
- Organisation