Incident Library

IL-0285 Clever Cloud · 2024-08-02databaseobservability

Clever Cloud's Pulsar brokers stalled in a minor upgrade, then hypervisors overloaded and crashed with kernel panics

Clever Cloud uses Apache Pulsar as its basic messaging layer, with proxies between customers' clients and the brokers. Minor releases are applied as rolling upgrades, during which topics move from one broker to another; because Pulsar brokers are stateless, this handover is normally seamless. In this maintenance the team also opted into the TransferShedder balancer and the extensible bundle unload strategy, which shares load metadata among brokers through non-persistent real-time replication: the first upgraded broker creates a system topic for it. The extension failed to initialise and never created the topic, but the broker kept running, so monitoring saw it as healthy and the rolling upgrade continued. Each further broker looked up the topic metadata and filled the lookup request queue, until no topics were loaded. Clever Cloud validates non-minor releases in a simulation of a full environment, but that process was not used for minor upgrades or small configuration changes.

Rolling back the unload strategy at 10:58 CEST did not help, because the system topic it had created was assigned to no broker and blocked every broker from starting. Engineers shut down the client proxies at 11:36, the deployments team switched notification events to RabbitMQ at 11:50 as a workaround, and at 12:22 all brokers were stopped and only one was started, which could load the topic so it could be deleted; all brokers were up at 12:28 and messaging worked again when the proxies restarted at 12:54. Meanwhile the Vector observability agent inside customer VMs had buffered logs in memory while Pulsar was unreachable. Small VMs hit their memory limit, the kernel dropped disk caches and reloaded processes from disk, and the resulting disk I/O, about 25Gbps, overloaded hypervisors across all availability zones. Clever Cloud sets the kernel's khungtaskd watchdog to panic when a task stays in TASK_UNINTERRUPTIBLE longer than the default 120s, so hypervisors with tasks stuck on disk writes or KVM calls panicked and rebooted. Monitoring read unreachable or slowed VMs as failures or CPU growth and ordered redeployments and upscales, customers redeployed by hand, and because Clever Cloud runs on its own platform, the control plane that had to schedule all this was itself slowed.

To regain control the team stopped the watchdog, disabled Metrics API ingestion and stopped the Warp10/FoundationDB long-term metrics storage, put all monitoring instances into maintenance, killed the overloaded small VMs one by one, drained hypervisors, added parked hypervisors to EU-FR-1, removed an overloaded RabbitMQ node, rolling-restarted every load balancer, restarted all HV agents and destroyed dangling VMs. Hypervisors were back to nominal load at 18:36 CEST, all deployments were rolling out from 18:20, and the impact ended at 22:53 when Metrics API ingestion was fully restored. EU-FR-1 (PAR) and remote zones that depend on its control plane (OVHcloud, Scaleway and Oracle) were affected; private and on-premise zones were not, and no data was lost.

Severity
–
Scope
The Deployment API was unreachable and deployments and scaling failed; applications and databases on impacted hypervisors were degraded or unavailable, some cluster services had issues, most VPN endpoints were unavailable and the observability service was down; no data was lost
Users affected
many Clever Cloud customers
Region
EU-FR-1 (PAR) and remote zones depending on the EU-FR-1 control plane (OVHcloud, Scaleway and Oracle)
Time to detect
2m
Time to resolve
12h 47m

Symptoms

On 2 August 2024, two minutes after Clever Cloud's data team started a minor rolling upgrade of its Apache Pulsar messaging cluster to release 3.3.1, monitoring showed connection problems with the messaging platform. Producing data into the event bus failed, the backlog grew while no new producers or consumers succeeded, and agents on hypervisors, log forwarders on load balancers and deployments were affected. At 10:20 CEST the Pulsar cluster was declared defective.

A broker logged "failed to get active owner broker" and "java.lang.IllegalStateException: The new owner clevercloud-pulsar-broker-c3-n1:8443 is inactive", but it did not crash, so monitoring saw it as up and the rolling upgrade went on. Clients then failed with "Failed due to too many pending lookup requests", monitoring showed the "Lookup request" queue full on all brokers (50k pending requests), and the system topic non-persistent://pulsar/system/loadbalancer-broker-load-data failed with "Failed to get partition metadata" and "The registry already closed." In the end no topics were loaded on any broker, and the team noticed the drop in bandwidth across the platform.

The failure spread from messaging to the whole platform. From 11:10 CEST customers could no longer deploy applications. After the brokers came back at 12:28, CPU load rose sharply on almost all hypervisors in all regions, the alert "HV Agent has too many tasks" started at 12:35 and hypervisors began killing VM management processes, some load balancers among them. The main API components crashed with out-of-memory (OOM) errors at 12:38, hypervisors rebooted from kernel panics from 12:47 on hung VM tasks in the KVM API or on ext4 writes, private networks flapped, the observability platform went down, and by 13:05 the Clever Cloud API and admin tools were down. Hypervisor disks reached about 25Gbps, and monitoring ordered almost three times the usual number of deployments that day.

At 10:58 CEST engineers understood that the new load balancer parameter conflicted with a transparent rolling upgrade and rolled it back, keeping release 3.3.1. The brokers then failed to restart because a system topic created by the new strategy had conflicting metadata and could not be loaded or deleted with admin commands. Later, seeing high CPU load and disk reads mostly from small pico and nano VMs, engineers suspected that Vector could not push VM logs to Pulsar and was pushing those VMs into swap. The watchdog that reloads failed components was stopped at 12:39 so that engineers could start components one by one.

Symptom Deploys failingSustained high node CPUMemory pressureNode kernel panicOOMKilledAutoscaler misbehaviourMonitoring outage

Root cause

Root cause Configuration change

Component Message brokerTelemetry agentLinux kernelAutomated capacity manager
Contributing factor No configuration testsMonitoring blind spotIneffective rollbackUnreviewed defaultsAutomation amplificationRetry stormCircular dependency
Trigger
At 10:06 CEST the data team began a minor rolling upgrade of Apache Pulsar to release 3.3.1 that also switched on a new balancer for bundle placement: the extensible load manager (loadManagerClassName=org.apache.pulsar.broker.loadbalance.extensions.ExtensibleLoadManagerImpl) with the TransferShedder load shedding strategy instead of the native OverloadShedder.

Detection

Monitoring showed connection issues to the messaging platform at 10:08 CEST, two minutes after the upgrade began, and engineers confirmed at 10:09 that producing data into the event bus failed. Monitoring did not catch the first broker's failure to create the system topic, because that broker stayed up.

Remediation

Action items

Lessons

The simulation process that validates non-minor releases was not applied to minor upgrades or to small configuration changes, and this one went straight to production. Running Clever Cloud on Clever Cloud turned into a chicken-and-egg situation: recovery needed the scheduler to carry out large actions while the control plane was itself slowed and the infrastructure had no free resources. Measuring each key indicator by time of event, detection, decision and resolution showed where the team had lacked visibility or reaction and challenged established rules.

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Message broker, Telemetry agent, Linux kernel among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.