Cases3 cases

1 to 3 of 3

1. IL-0560 Some PubNub messages in the US East point of presence were misrouted with no error and not delivered right away

   PubNub2026

   Symptoms

   At 12:10 UTC on 25 August 2026, PubNub's internal monitoring alerted that a very small subset of messages in one availability zone of the US East point of presence might not have been delivered right away to subscribers in the same region. Publish and Subscribe calls returned no errors, even though some messages did not appear in the responses.

   Misrouted requestsInconsistent reads from a data store

   Trigger

   During planned internal load testing, a group of internal routing servers was removed from service.

   Root cause

   Configuration change

   Component

   Service discovery

   Contributing factor

   –

   Detection

   PubNub's internal monitoring alerted at 12:10 UTC; synthetic monitoring then confirmed that no more messages were being missed.

   Remediation

   • Performed a rolling restart of the affected servers to clear the outdated routing addresses

   Action items

   • Deploy a fix to all publishing servers worldwide so that outdated internal routing addresses are retired correctly

   View case
2. IL-0559 PubNub services degraded in US-East during an AWS outage in the IAD region, and failover load then degraded US-West

   PubNub2025

   Symptoms

   At 07:06 UTC on 20 October 2025, PubNub's monitoring alerted on raised error levels across multiple services in the IAD (US-East) region. Customers could see higher error rates and latency, and Presence was intermittently unavailable in IAD, SJC (US-West) and HND (AP-Northeast).

   High error rateHigh latency

   Trigger

   A broad infrastructure outage at AWS, PubNub's cloud provider, in the IAD (US-East) region.

   Root cause

   Cloud provider outage

   Component

   –

   Contributing factor

   No runbookCircular dependencyNo headroom

   Detection

   PubNub's monitoring systems alerted at 07:06 UTC on raised error levels across multiple services in IAD.

   Remediation

   • Started regional failover procedures and re-routed new connections to other regions
   • Reset established connections by hand
   • Re-routed Presence traffic to Frankfurt (EU-Central)
   • Brought on additional infrastructure in other regions to absorb traffic
   • Isolated the US-East region again when the second wave hit US-West
   • Scaled up balancer capacity in US-West
   • Rebalanced traffic after AWS restored service and US-East was confirmed stable

   Action items

   • Consolidate and centralise regional failover procedures so they are complete and immediately accessible for all production services
   • Fill gaps in process documentation for newer services before they are fully adopted into production
   • Fix internal tooling problems, including inventory and DNS resolution, that made mitigation harder

   View case
3. IL-0558 New PubNub instance types in FRA hit connectivity limits, and Pub/Sub, History and Presence saw intermittent errors

   PubNub2025

   Symptoms

   From 08:15 UTC on Friday 27 June 2025, PubNub's Pub/Sub, History and Presence services showed occasional, intermittent increases in latency and server errors in the Europe (FRA) region.

   High latencyHigh error rate

   Trigger

   PubNub introduced new instance types to its system to get access to more cloud server capacity across its regions.

   Root cause

   Capacity exhaustion

   Component

   Cloud load balancer

   Contributing factor

   Partial rolloutUnverified assumptionMonitoring blind spot

   Remediation

   • Configured the affected services to run on other instance types
   • Launched more capacity

   Action items

   • Make the new instance types emit metrics on OS thresholds and limits, so approaching or exceeding a limit is detected on any instance type
   • Scale ahead of demand and route traffic according to instance type

   View case

By root cause3