Cases19 cases

2 to 6 of 19

1. IL-0559 PubNub services degraded in US-East during an AWS outage in the IAD region, and failover load then degraded US-West

   PubNub2025

   Symptoms

   At 07:06 UTC on 20 October 2025, PubNub's monitoring alerted on raised error levels across multiple services in the IAD (US-East) region. Customers could see higher error rates and latency, and Presence was intermittently unavailable in IAD, SJC (US-West) and HND (AP-Northeast).

   High error rateHigh latency

   Trigger

   A broad infrastructure outage at AWS, PubNub's cloud provider, in the IAD (US-East) region.

   Root cause

   Cloud provider outage

   Component

   –

   Contributing factor

   No runbookCircular dependencyNo headroom

   Detection

   PubNub's monitoring systems alerted at 07:06 UTC on raised error levels across multiple services in IAD.

   Remediation

   • Started regional failover procedures and re-routed new connections to other regions
   • Reset established connections by hand
   • Re-routed Presence traffic to Frankfurt (EU-Central)
   • Brought on additional infrastructure in other regions to absorb traffic
   • Isolated the US-East region again when the second wave hit US-West
   • Scaled up balancer capacity in US-West
   • Rebalanced traffic after AWS restored service and US-East was confirmed stable

   Action items

   • Consolidate and centralise regional failover procedures so they are complete and immediately accessible for all production services
   • Fill gaps in process documentation for newer services before they are fully adopted into production
   • Fix internal tooling problems, including inventory and DNS resolution, that made mitigation harder

   View case
2. IL-0284 Workers KV failed 90.22% of requests in a third-party storage outage, breaking Access, WARP and many Cloudflare products

   Cloudflare2025

   Symptoms

   At 17:52 UTC on 12 June 2025, Cloudflare's WARP team saw registrations of new devices failing and declared an incident. Minutes later the Cloudflare Access team got an alert on a rapid rise in error rates, and Service Level Objectives for several other services fell below target. All of them shared one cause: Workers KV was unavailable.

   High error rateService outageCredentials unavailable

   Trigger

   Part of the storage infrastructure behind Workers KV is backed by a third-party cloud provider, and that provider had an outage. Workers KV could then no longer serve cold reads or writes from its central data store.

   Root cause

   Cloud provider outage

   Component

   Database

   Contributing factor

   Single point of failureExcessive blast radiusCircular dependencyRetry storm

   Detection

   The WARP team saw registrations of new devices failing at 17:52 UTC and declared an incident. At 18:05 the Access team received an alert on a rapid increase in error rates and Service Level Objective alerts fired across several teams; at 18:06 the incidents were combined once Workers KV unavailability was identified as the shared cause.

   Remediation

   • Activated the Challenges and Turnstile kill switches so visitors were not blocked
   • Degraded Gateway rules that referenced identity or device posture state to remove the Workers KV dependency
   • Force-dropped Access identity and device posture requests to shed load on Workers KV
   • Worked on a Workers KV release against an alternative backing datastore in case the storage stayed down
   • Restored Access and Device Posture calls to Workers KV once the third-party service was back

   Action items

   • Avoid singular dependencies on storage infrastructure that Cloudflare does not own
   • Improve the ability to recover critical services, including Access, Gateway and WARP
   • Add redundancy to Workers KV's storage infrastructure and remove the dependency on any single provider
   • Make each affected product resilient to the loss of any single point of failure, third-party dependencies included
   • Build tooling to re-enable namespaces progressively during storage incidents, so services come back without overloading Cloudflare's own infrastructure

   Lessons

   Cloudflare holds itself responsible for its choice of dependencies and how it architects around them, even when a vendor failure was the trigger. Building many products on Workers KV normally makes them more robust than separate storage services would, but here the cascading failure of Workers KV made the outage worse and widened its blast radius.

   View case
3. IL-0227 Most Heroku dynos in Private Spaces lost outbound connectivity after an automated OS update restarted host networking

   Heroku2025

   Symptoms

   From 06:00 UTC on 10 June 2025, Heroku services degraded badly: customers reported intermittent logins, applications failed, and Heroku's monitoring detected widespread errors with dyno networking. Heroku's own tools and the Heroku Status Page were affected as well, which delayed communication with customers and left many of them without accurate status updates. Only at 13:58 UTC did the team find a workaround to post updates to the @herokustatus account on X.

   Network unreachableService outageHigh error rate

   Trigger

   An automated operating system update ran on Heroku's production infrastructure when it should have been disabled, and it restarted the hosts' networking services.

   Root cause

   Breaking change in an upgrade

   Component

   Host routes and routing rules

   Contributing factor

   Unpinned versionCircular dependencyLimited debug access

   Detection

   Customers reported intermittent logins, and Heroku's monitoring detected widespread errors with dyno networking. Heroku's own tools and the Heroku Status Page were affected as well.

   Remediation

   • Restarted internal instances in rotation, which gave some relief
   • Had an upstream vendor invalidate the token used for the automated updates
   • Disabled the automated upgrade service
   • Recycled dynos across the whole fleet

   Action items

   • Put permanent controls in place so no system change happens outside the controlled deployment process
   • Audit all base images for similar risks
   • Make network routing survive graceful restarts of the networking service
   • Build fully independent backup communication channels
   • Overhaul incident response tooling, including tools that run queries across the whole fleet
   • Streamline break-glass procedures for rapid access to critical systems
   • Improve monitoring to detect complex issues earlier

   Lessons

   Heroku's post-mortem names three areas: production lacked immutability controls, so an automated process could change it; communication was too slow, made worse by a status page that went down with the platform; and gaps in tooling and process made recovery take longer than it should have.

   View case
4. IL-0543 Twingate API traffic piled onto one backend behind the global load balancer, failing core services for 29 minutes

   Twingate2025

   Symptoms

   At 20:45 UTC on 28 May 2025, most of Twingate's API traffic started going to a single backend service, which was overwhelmed and returned elevated error rates, so API calls failed widely across core services.

   Misrouted requestsLoad imbalanceHigh error rate

   Trigger

   A rollout that updated the routing rules of Twingate's global load balancer.

   Root cause

   Configuration change

   Component

   Cloud load balancer

   Contributing factor

   Unrepresentative stagingCircular dependency

   Detection

   The incident was detected at 20:48 UTC, three minutes after API traffic was first misrouted, and triage began.

   Remediation

   • Started a rollback of the routing change at 20:52 UTC; the automated rollback stalled on infrastructure dependencies
   • Applied a manual fix with local tooling at 21:09 UTC, which restored service by 21:14 UTC

   Action items

   • Correct the routing rule
   • Add alerts that detect backend traffic anomalies in lower environments
   • Improve resource scaling thresholds
   • Decouple infrastructure tooling from service dependencies so recovery has faster options
   • Introduce backend sharding to isolate customer workloads and reduce the impact of incidents

   View case
5. IL-0285 Clever Cloud's Pulsar brokers stalled in a minor upgrade, then hypervisors overloaded and crashed with kernel panics

   Clever Cloud2024

   Symptoms

   On 2 August 2024, two minutes after Clever Cloud's data team started a minor rolling upgrade of its Apache Pulsar messaging cluster to release 3.3.1, monitoring showed connection problems with the messaging platform. Producing data into the event bus failed, the backlog grew while no new producers or consumers succeeded, and agents on hypervisors, log forwarders on load balancers and deployments were affected. At 10:20 CEST the Pulsar cluster was declared defective.

   Deploys failingSustained high node CPUMemory pressureNode kernel panicOOMKilledAutoscaler misbehaviourMonitoring outage

   Trigger

   At 10:06 CEST the data team began a minor rolling upgrade of Apache Pulsar to release 3.3.1 that also switched on a new balancer for bundle placement: the extensible load manager (loadManagerClassName=org.apache.pulsar.broker.loadbalance.extensions.ExtensibleLoadManagerImpl) with the TransferShedder load shedding strategy instead of the native OverloadShedder.

   Root cause

   Configuration change

   Component

   Message brokerTelemetry agentLinux kernelAutomated capacity manager

   Contributing factor

   No configuration testsMonitoring blind spotIneffective rollbackUnreviewed defaultsAutomation amplificationRetry stormCircular dependency

   Detection

   Monitoring showed connection issues to the messaging platform at 10:08 CEST, two minutes after the upgrade began, and engineers confirmed at 10:09 that producing data into the event bus failed. Monitoring did not catch the first broker's failure to create the system topic, because that broker stayed up.

   Remediation

   • Rolled back the new unload strategy while keeping Pulsar 3.3.1
   • Shut down the client proxies to release the brokers
   • Switched deployment notification events to RabbitMQ as a workaround
   • Stopped all brokers and started only one, which loaded the conflicting system topic so it could be deleted, then started all brokers
   • Restarted the messaging proxies, after which messages flowed again
   • Stopped the watchdog so it would not reload every failed component, and started components one by one
   • Disabled Metrics API ingestion to protect its storage cluster
   • Stopped the long-term storage for observability metrics (the Warp10/FoundationDB cluster) to free resources
   • Put all monitoring instances in all regions into maintenance so they stopped sending events to the orchestrator
   • Killed the overloaded small VMs one by one
   • Flagged 12 hypervisors to drain workload and shut down some load balancers to free CPU and memory
   • Cleaned zombie VMs to speed up recovery
   • Cleared the monitoring alert queue so queued events would not flood the orchestrator
   • Added parked hypervisors to EU-FR-1 to relieve the overloaded ones
   • Removed an overloaded RabbitMQ node from the cluster used by the orchestrator
   • Rolling-restarted every regional L7 load balancer and every L4 load balancer
   • Restarted all HV agents and the add-on provider APIs
   • Destroyed all dangling VMs stuck in the stopping state
   • Re-enabled Metrics API ingestion with a ramp-up QoS policy

   Action items

   • Track KPIs for Pulsar broker and cluster health, hypervisor load, I/O and kernel panic reboots, zombie VMs, RabbitMQ broker and cluster health and deployment queue consumption, each by time of event, detection, decision and resolution
   • Deal with too many monitoring alerts
   • Look at a backup bastion, because a regional bastion is vulnerable
   • Rework the control plane recovery process
   • Rework the correlation between monitoring and orchestration
   • Revisit fault domain separation against multi-tenancy
   • Rethink runtime images and the role of the hypervisor
   • Isolate waved release upgrades of Pulsar
   • Redesign runtime images with better hypervisor integration
   • Regionalise observability for redundancy
   • Set Vector's buffer.max_events and buffer.max_size for small VMs
   • Switch Vector's buffer.type from memory, the default, to disk
   • Change Vector's buffer.when_full from block, the default, to drop_newest
   • Tune journald verbosity and SystemMaxUse on small VMs
   • Add cgroups restrictions for better process control
   • Add I/O restrictions for the VM manager
   • Detect zombie VMs better

   Lessons

   The simulation process that validates non-minor releases was not applied to minor upgrades or to small configuration changes, and this one went straight to production. Running Clever Cloud on Clever Cloud turned into a chicken-and-egg situation: recovery needed the scheduler to carry out large actions while the control plane was itself slowed and the infrastructure had no free resources. Measuring each key indicator by time of event, detection, decision and resolution showed where the team had lacked visibility or reaction and challenged established rules.

   View case

By root cause8