Incident Library
Applied filters ComponentTransit gateway or network interconnectSymptomHigh latencyClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases3 cases

1 to 3 of 3

  1. IL-0341 Traffic between Cloudflare and AWS us-east-1 had high latency, packet loss and origin failures for almost four hours
    Cloudflare2025
    Symptoms

    From 16:27 UTC on 21 August 2025, users connecting to Cloudflare from servers in AWS us-east-1, or receiving connections from Cloudflare there, saw high latency, packet loss and failures to reach origins. Cloudflare's network team was alerted to internal congestion in Ashburn (IAD) at 16:44 UTC.

    Packet lossHigh latencyIntermittent timeouts
    Trigger

    At about 16:27 UTC a single customer began sending many requests from AWS us-east-1 to Cloudflare for objects in Cloudflare's cache, doubling the total traffic from Cloudflare to AWS.

    Root cause
    Capacity exhaustion
    Component
    Transit gateway or network interconnectBGP routingCDN and edge network
    Contributing factor
    No headroomHarmful manual actionExcessive blast radius
    Detection

    Cloudflare's network team was alerted to internal congestion in Ashburn (IAD) at 16:44 UTC, and an incident was raised for customer impact at 17:45 UTC.

    Remediation
    • Rate limited the single customer whose traffic caused the surge
    • Took further traffic engineering actions that fully resolved the congestion
    • Asked AWS to revert its BGP prefix withdrawals, which AWS finished at 20:07 UTC
    Action items
    • Build a mechanism to deprioritise a customer's traffic when it congests the network enough to affect others
    • Speed up the Data Center Interconnect upgrades to add network capacity
    • Work with AWS so the two companies' BGP traffic engineering actions do not conflict
    • Build a traffic management system that gives each customer a network resource budget and automates the manual actions used here
    Lessons

    Cloudflare concludes that it must isolate customers better so that no single customer can take over shared network resources and degrade the platform for others, and that its network needs enough capacity to meet demand. It describes the outage as the result of insufficient network congestion management during an unusual high-traffic event.

    View case
  2. IL-0223 A connection surge congested the devices between AWS's internal and main networks in us-east-1, breaking EC2 APIs
    Amazon Web Services2021
    Symptoms

    From 7:30 AM PST on 7 December 2021, AWS services in us-east-1 that communicate between AWS's internal network and its main network saw rising latency and errors. The AWS operations teams lost real-time monitoring data at once, and the first thing found in the logs was elevated internal DNS errors.

    Control plane lagHigh latencyHigh error rateMonitoring outageDNS resolution failures
    Trigger

    At 7:30 AM PST an automated activity that scales the capacity of one AWS service hosted in the main AWS network triggered unexpected behaviour from a large number of clients inside the internal network.

    Root cause
    Software defect
    Component
    Transit gateway or network interconnectAutomated capacity manager
    Contributing factor
    Retry stormCircular dependency
    Detection

    The congestion immediately cut off real-time monitoring data for the AWS operations teams, who turned to logs and first identified elevated internal DNS errors.

    Remediation
    • Moved internal DNS traffic away from the congested network paths
    • Isolated the top sources of traffic onto dedicated network devices
    • Disabled some heavy network traffic services, including EventBridge event delivery
    • Brought additional networking capacity online
    • Disabled the automated scaling activities that triggered the event
    Action items
    • Keep the triggering scaling activities off until all remediations are deployed
    • Fix the latent issue that kept networking clients from backing off
    • Deploy network configuration that protects the affected networking devices in a similar congestion event
    • Keep API Gateway servers healthy when the internal network is unavailable and speed up their recycling
    • Release a new Service Health Dashboard that makes service impact easier to understand
    • Run the support system actively across multiple AWS regions
    View case
  3. IL-0362 PagerDuty could not dispatch notifications for 18 minutes when a peering point shared by two AWS regions degraded
    PagerDuty2013sev1
    Symptoms

    On Saturday 13 April 2013 at 8:11am Pacific Time, PagerDuty's on-call engineer was paged about one node of the notification dispatch system, and an attempt at 8:13 to bring the node back failed. Customers saw delayed notifications and 500 errors on PagerDuty's API endpoints.

    High latencyHigh error rateIntermittent connection failures
    Trigger

    At 7:57am Pacific Time, according to AWS, a peering point in Northern California began to degrade, causing connectivity problems for the two AWS regions that PagerDuty used as two of its three datacenters.

    Root cause
    Cloud provider outage
    Component
    Transit gateway or network interconnectCoordination and key-value store
    Contributing factor
    Single point of failureUnverified assumptionMonitoring blind spot
    Detection

    The PagerDuty on-call engineer was paged at 8:11am about an issue with one node of the notification dispatch system, and at 8:18 monitoring detected multiple-provider failures for notifications.

    Remediation
    • Restarted most of the failed coordinator processes by hand
    • The notification dispatch system reached quorum again and processed all queued notifications
    Action items
    • Add more logging and aggregate logs into a single searchable source
    • Add a process watcher that restarts failed coordinator processes automatically
    • Build a dashboard that shows inter-host connectivity
    • Train engineering staff on Cassandra and ZooKeeper
    • Investigate moving off one of the AWS regions to a new hosting provider and datacenter that avoids a single point of failure
    Lessons

    Datacenters chosen to be independent and physically separate can still share a network peering point, and a design built to survive the loss of one datacenter did not survive losing two at once.

    View case
Check your cluster for this

These 3 cases involve Transit gateway or network interconnect and BGP routing. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause3

Capacity exhaustion1 case

Component
BGP routingCDN and edge networkTransit gateway or network interconnect
Contributing factor
Excessive blast radiusHarmful manual actionNo headroom
Organisation
Cloudflare
  1. IL-0341 Traffic between Cloudflare and AWS us-east-1 had high latency, packet loss and origin failures for almost four hours
    Cloudflare2025

Software defect1 case

Component
Automated capacity managerTransit gateway or network interconnect
Contributing factor
Circular dependencyRetry storm
Organisation
Amazon Web Services
  1. IL-0223 A connection surge congested the devices between AWS's internal and main networks in us-east-1, breaking EC2 APIs
    Amazon Web Services2021

Cloud provider outage1 case

Component
Coordination and key-value storeTransit gateway or network interconnect
Contributing factor
Unverified assumptionMonitoring blind spotSingle point of failure
Organisation
PagerDuty
  1. IL-0362 PagerDuty could not dispatch notifications for 18 minutes when a peering point shared by two AWS regions degraded
    PagerDuty2013