Cases1 case

1 to 1 of 1

1. IL-0559 PubNub services degraded in US-East during an AWS outage in the IAD region, and failover load then degraded US-West

   PubNub2025

   Symptoms

   At 07:06 UTC on 20 October 2025, PubNub's monitoring alerted on raised error levels across multiple services in the IAD (US-East) region. Customers could see higher error rates and latency, and Presence was intermittently unavailable in IAD, SJC (US-West) and HND (AP-Northeast).

   High error rateHigh latency

   Trigger

   A broad infrastructure outage at AWS, PubNub's cloud provider, in the IAD (US-East) region.

   Root cause

   Cloud provider outage

   Component

   –

   Contributing factor

   No runbookCircular dependencyNo headroom

   Detection

   PubNub's monitoring systems alerted at 07:06 UTC on raised error levels across multiple services in IAD.

   Remediation

   • Started regional failover procedures and re-routed new connections to other regions
   • Reset established connections by hand
   • Re-routed Presence traffic to Frankfurt (EU-Central)
   • Brought on additional infrastructure in other regions to absorb traffic
   • Isolated the US-East region again when the second wave hit US-West
   • Scaled up balancer capacity in US-West
   • Rebalanced traffic after AWS restored service and US-East was confirmed stable

   Action items

   • Consolidate and centralise regional failover procedures so they are complete and immediately accessible for all production services
   • Fill gaps in process documentation for newer services before they are fully adopted into production
   • Fix internal tooling problems, including inventory and DNS resolution, that made mitigation harder

   View case

By root cause1