Cases10 cases

1 to 5 of 10

1. IL-0559 PubNub services degraded in US-East during an AWS outage in the IAD region, and failover load then degraded US-West

   PubNub2025

   Symptoms

   At 07:06 UTC on 20 October 2025, PubNub's monitoring alerted on raised error levels across multiple services in the IAD (US-East) region. Customers could see higher error rates and latency, and Presence was intermittently unavailable in IAD, SJC (US-West) and HND (AP-Northeast).

   High error rateHigh latency

   Trigger

   A broad infrastructure outage at AWS, PubNub's cloud provider, in the IAD (US-East) region.

   Root cause

   Cloud provider outage

   Component

   –

   Contributing factor

   No runbookCircular dependencyNo headroom

   Detection

   PubNub's monitoring systems alerted at 07:06 UTC on raised error levels across multiple services in IAD.

   Remediation

   • Started regional failover procedures and re-routed new connections to other regions
   • Reset established connections by hand
   • Re-routed Presence traffic to Frankfurt (EU-Central)
   • Brought on additional infrastructure in other regions to absorb traffic
   • Isolated the US-East region again when the second wave hit US-West
   • Scaled up balancer capacity in US-West
   • Rebalanced traffic after AWS restored service and US-East was confirmed stable

   Action items

   • Consolidate and centralise regional failover procedures so they are complete and immediately accessible for all production services
   • Fill gaps in process documentation for newer services before they are fully adopted into production
   • Fix internal tooling problems, including inventory and DNS resolution, that made mitigation harder

   View case
2. IL-0284 Workers KV failed 90.22% of requests in a third-party storage outage, breaking Access, WARP and many Cloudflare products

   Cloudflare2025

   Symptoms

   At 17:52 UTC on 12 June 2025, Cloudflare's WARP team saw registrations of new devices failing and declared an incident. Minutes later the Cloudflare Access team got an alert on a rapid rise in error rates, and Service Level Objectives for several other services fell below target. All of them shared one cause: Workers KV was unavailable.

   High error rateService outageCredentials unavailable

   Trigger

   Part of the storage infrastructure behind Workers KV is backed by a third-party cloud provider, and that provider had an outage. Workers KV could then no longer serve cold reads or writes from its central data store.

   Root cause

   Cloud provider outage

   Component

   Database

   Contributing factor

   Single point of failureExcessive blast radiusCircular dependencyRetry storm

   Detection

   The WARP team saw registrations of new devices failing at 17:52 UTC and declared an incident. At 18:05 the Access team received an alert on a rapid increase in error rates and Service Level Objective alerts fired across several teams; at 18:06 the incidents were combined once Workers KV unavailability was identified as the shared cause.

   Remediation

   • Activated the Challenges and Turnstile kill switches so visitors were not blocked
   • Degraded Gateway rules that referenced identity or device posture state to remove the Workers KV dependency
   • Force-dropped Access identity and device posture requests to shed load on Workers KV
   • Worked on a Workers KV release against an alternative backing datastore in case the storage stayed down
   • Restored Access and Device Posture calls to Workers KV once the third-party service was back

   Action items

   • Avoid singular dependencies on storage infrastructure that Cloudflare does not own
   • Improve the ability to recover critical services, including Access, Gateway and WARP
   • Add redundancy to Workers KV's storage infrastructure and remove the dependency on any single provider
   • Make each affected product resilient to the loss of any single point of failure, third-party dependencies included
   • Build tooling to re-enable namespaces progressively during storage incidents, so services come back without overloading Cloudflare's own infrastructure

   Lessons

   Cloudflare holds itself responsible for its choice of dependencies and how it architects around them, even when a vendor failure was the trigger. Building many products on Workers KV normally makes them more robust than separate storage services would, but here the cascading failure of Workers KV made the outage worse and widened its blast radius.

   View case
3. IL-0227 Most Heroku dynos in Private Spaces lost outbound connectivity after an automated OS update restarted host networking

   Heroku2025

   Symptoms

   From 06:00 UTC on 10 June 2025, Heroku services degraded badly: customers reported intermittent logins, applications failed, and Heroku's monitoring detected widespread errors with dyno networking. Heroku's own tools and the Heroku Status Page were affected as well, which delayed communication with customers and left many of them without accurate status updates. Only at 13:58 UTC did the team find a workaround to post updates to the @herokustatus account on X.

   Network unreachableService outageHigh error rate

   Trigger

   An automated operating system update ran on Heroku's production infrastructure when it should have been disabled, and it restarted the hosts' networking services.

   Root cause

   Breaking change in an upgrade

   Component

   Host routes and routing rules

   Contributing factor

   Unpinned versionCircular dependencyLimited debug access

   Detection

   Customers reported intermittent logins, and Heroku's monitoring detected widespread errors with dyno networking. Heroku's own tools and the Heroku Status Page were affected as well.

   Remediation

   • Restarted internal instances in rotation, which gave some relief
   • Had an upstream vendor invalidate the token used for the automated updates
   • Disabled the automated upgrade service
   • Recycled dynos across the whole fleet

   Action items

   • Put permanent controls in place so no system change happens outside the controlled deployment process
   • Audit all base images for similar risks
   • Make network routing survive graceful restarts of the networking service
   • Build fully independent backup communication channels
   • Overhaul incident response tooling, including tools that run queries across the whole fleet
   • Streamline break-glass procedures for rapid access to critical systems
   • Improve monitoring to detect complex issues earlier

   Lessons

   Heroku's post-mortem names three areas: production lacked immutability controls, so an automated process could change it; communication was too slow, made worse by a status page that went down with the platform; and gaps in tooling and process made recovery take longer than it should have.

   View case
4. IL-0543 Twingate API traffic piled onto one backend behind the global load balancer, failing core services for 29 minutes

   Twingate2025

   Symptoms

   At 20:45 UTC on 28 May 2025, most of Twingate's API traffic started going to a single backend service, which was overwhelmed and returned elevated error rates, so API calls failed widely across core services.

   Misrouted requestsLoad imbalanceHigh error rate

   Trigger

   A rollout that updated the routing rules of Twingate's global load balancer.

   Root cause

   Configuration change

   Component

   Cloud load balancer

   Contributing factor

   Unrepresentative stagingCircular dependency

   Detection

   The incident was detected at 20:48 UTC, three minutes after API traffic was first misrouted, and triage began.

   Remediation

   • Started a rollback of the routing change at 20:52 UTC; the automated rollback stalled on infrastructure dependencies
   • Applied a manual fix with local tooling at 21:09 UTC, which restored service by 21:14 UTC

   Action items

   • Correct the routing rule
   • Add alerts that detect backend traffic anomalies in lower environments
   • Improve resource scaling thresholds
   • Decouple infrastructure tooling from service dependencies so recovery has faster options
   • Introduce backend sharding to isolate customer workloads and reduce the impact of incidents

   View case
5. IL-0223 A connection surge congested the devices between AWS's internal and main networks in us-east-1, breaking EC2 APIs

   Amazon Web Services2021

   Symptoms

   From 7:30 AM PST on 7 December 2021, AWS services in us-east-1 that communicate between AWS's internal network and its main network saw rising latency and errors. The AWS operations teams lost real-time monitoring data at once, and the first thing found in the logs was elevated internal DNS errors.

   Control plane lagHigh latencyHigh error rateMonitoring outageDNS resolution failures

   Trigger

   At 7:30 AM PST an automated activity that scales the capacity of one AWS service hosted in the main AWS network triggered unexpected behaviour from a large number of clients inside the internal network.

   Root cause

   Software defect

   Component

   Transit gateway or network interconnectAutomated capacity manager

   Contributing factor

   Retry stormCircular dependency

   Detection

   The congestion immediately cut off real-time monitoring data for the AWS operations teams, who turned to logs and first identified elevated internal DNS errors.

   Remediation

   • Moved internal DNS traffic away from the congested network paths
   • Isolated the top sources of traffic onto dedicated network devices
   • Disabled some heavy network traffic services, including EventBridge event delivery
   • Brought additional networking capacity online
   • Disabled the automated scaling activities that triggered the event

   Action items

   • Keep the triggering scaling activities off until all remediations are deployed
   • Fix the latent issue that kept networking clients from backing off
   • Deploy network configuration that protects the affected networking devices in a similar congestion event
   • Keep API Gateway servers healthy when the internal network is unavailable and speed up their recycling
   • Release a new Service Health Dashboard that makes service impact easier to understand
   • Run the support system actively across multiple AWS regions

   View case

By root cause5