Incident Library
Applied filters Root causeCloud provider outageSymptomHigh latencyFactorNo runbookClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0559 PubNub services degraded in US-East during an AWS outage in the IAD region, and failover load then degraded US-West
    PubNub2025
    Symptoms

    At 07:06 UTC on 20 October 2025, PubNub's monitoring alerted on raised error levels across multiple services in the IAD (US-East) region. Customers could see higher error rates and latency, and Presence was intermittently unavailable in IAD, SJC (US-West) and HND (AP-Northeast).

    High error rateHigh latency
    Trigger

    A broad infrastructure outage at AWS, PubNub's cloud provider, in the IAD (US-East) region.

    Root cause
    Cloud provider outage
    Component
    –
    Contributing factor
    No runbookCircular dependencyNo headroom
    Detection

    PubNub's monitoring systems alerted at 07:06 UTC on raised error levels across multiple services in IAD.

    Remediation
    • Started regional failover procedures and re-routed new connections to other regions
    • Reset established connections by hand
    • Re-routed Presence traffic to Frankfurt (EU-Central)
    • Brought on additional infrastructure in other regions to absorb traffic
    • Isolated the US-East region again when the second wave hit US-West
    • Scaled up balancer capacity in US-West
    • Rebalanced traffic after AWS restored service and US-East was confirmed stable
    Action items
    • Consolidate and centralise regional failover procedures so they are complete and immediately accessible for all production services
    • Fill gaps in process documentation for newer services before they are fully adopted into production
    • Fix internal tooling problems, including inventory and DNS resolution, that made mitigation harder
    View case
Check your cluster for this

This case involves conditions a cluster can be checked for. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Cloud provider outage1 case

Component
–
Contributing factor
Circular dependencyNo runbookNo headroom
Organisation
PubNub
  1. IL-0559 PubNub services degraded in US-East during an AWS outage in the IAD region, and failover load then degraded US-West
    PubNub2025