Cases2 cases

1 to 2 of 2

1. IL-0559 PubNub services degraded in US-East during an AWS outage in the IAD region, and failover load then degraded US-West

   PubNub2025

   Symptoms

   At 07:06 UTC on 20 October 2025, PubNub's monitoring alerted on raised error levels across multiple services in the IAD (US-East) region. Customers could see higher error rates and latency, and Presence was intermittently unavailable in IAD, SJC (US-West) and HND (AP-Northeast).

   High error rateHigh latency

   Trigger

   A broad infrastructure outage at AWS, PubNub's cloud provider, in the IAD (US-East) region.

   Root cause

   Cloud provider outage

   Component

   –

   Contributing factor

   No runbookCircular dependencyNo headroom

   Detection

   PubNub's monitoring systems alerted at 07:06 UTC on raised error levels across multiple services in IAD.

   Remediation

   • Started regional failover procedures and re-routed new connections to other regions
   • Reset established connections by hand
   • Re-routed Presence traffic to Frankfurt (EU-Central)
   • Brought on additional infrastructure in other regions to absorb traffic
   • Isolated the US-East region again when the second wave hit US-West
   • Scaled up balancer capacity in US-West
   • Rebalanced traffic after AWS restored service and US-East was confirmed stable

   Action items

   • Consolidate and centralise regional failover procedures so they are complete and immediately accessible for all production services
   • Fill gaps in process documentation for newer services before they are fully adopted into production
   • Fix internal tooling problems, including inventory and DNS resolution, that made mitigation harder

   View case
2. IL-0558 New PubNub instance types in FRA hit connectivity limits, and Pub/Sub, History and Presence saw intermittent errors

   PubNub2025

   Symptoms

   From 08:15 UTC on Friday 27 June 2025, PubNub's Pub/Sub, History and Presence services showed occasional, intermittent increases in latency and server errors in the Europe (FRA) region.

   High latencyHigh error rate

   Trigger

   PubNub introduced new instance types to its system to get access to more cloud server capacity across its regions.

   Root cause

   Capacity exhaustion

   Component

   Cloud load balancer

   Contributing factor

   Partial rolloutUnverified assumptionMonitoring blind spot

   Remediation

   • Configured the affected services to run on other instance types
   • Launched more capacity

   Action items

   • Make the new instance types emit metrics on OS thresholds and limits, so approaching or exceeding a limit is detected on any instance type
   • Scale ahead of demand and route traffic according to instance type

   View case

By root cause2