IL-0559 PubNub · 2025-10-20awsnetworking

PubNub services degraded in US-East during an AWS outage in the IAD region, and failover load then degraded US-West

On 20 October 2025 an infrastructure outage at AWS in the IAD region raised error levels across multiple PubNub services from 07:06 UTC. PubNub started its regional failover procedures and moved new connections to other regions. Existing connections for some services stayed degraded because some failover processes had undefined steps and some tools were slow to reach during the provider issue. To restore service, engineers reset established connections by hand, moved Presence traffic to Frankfurt (EU-Central) and added infrastructure in other regions to take the traffic, and errors were mitigated by 09:20 UTC.

Later in the day the additional regional load in US-West set off a new wave of latency and errors there, reported from 12:40 UTC. PubNub isolated the US-East region again and scaled up balancer capacity in US-West, and services were stable by 13:20 UTC. AWS reported full restoration at 22:35 UTC; after checking that US-East was stable, PubNub finished rebalancing traffic by 23:48 UTC and declared the incident resolved.

Severity: –
Scope: Multiple PubNub services, including Publish/Subscribe, Presence and Access Manager, and the North America Points of Presence: higher error rates and latency, intermittent Presence availability
Users affected: –
Region: IAD (US-East), SJC (US-West), HND (AP-Northeast)
Time to detect: –
Time to resolve: 16h 42m

Symptoms

At 07:06 UTC on 20 October 2025, PubNub's monitoring alerted on raised error levels across multiple services in the IAD (US-East) region. Customers could see higher error rates and latency, and Presence was intermittently unavailable in IAD, SJC (US-West) and HND (AP-Northeast).

PubNub failed over regionally and re-routed new connections to other regions, but connections that were already open stayed degraded for some services for longer than expected. Some failover processes had undefined steps, some tools were slow to reach because of the provider's outage, and internal tooling had inventory and DNS resolution problems that made mitigation harder.

Errors were mitigated by 09:20 UTC. From 12:40 UTC the status page reported increased latency and errors for multiple services in US-West, a second wave caused by additional regional load there, and services were stable again at 13:20 UTC.

Symptom High error rate High latency

Root cause

Root cause Cloud provider outage

Component –

Contributing factor No runbook Circular dependency No headroom

Trigger: A broad infrastructure outage at AWS, PubNub's cloud provider, in the IAD (US-East) region.

Detection

PubNub's monitoring systems alerted at 07:06 UTC on raised error levels across multiple services in IAD.

Remediation

Started regional failover procedures and re-routed new connections to other regions
Reset established connections by hand
Re-routed Presence traffic to Frankfurt (EU-Central)
Brought on additional infrastructure in other regions to absorb traffic
Isolated the US-East region again when the second wave hit US-West
Scaled up balancer capacity in US-West
Rebalanced traffic after AWS restored service and US-East was confirmed stable

Action items

Consolidate and centralise regional failover procedures so they are complete and immediately accessible for all production services
Fill gaps in process documentation for newer services before they are fully adopted into production
Fix internal tooling problems, including inventory and DNS resolution, that made mitigation harder

Lessons

–

Sources

https://status.pubnub.com/incidents/4vl9wztw0cbn · official-status · archived copy

Related cases

–

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0559
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0559/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.