Incident Library
Applied filters ComponentTransit gateway or network interconnectSymptomIntermittent timeoutsClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0341 Traffic between Cloudflare and AWS us-east-1 had high latency, packet loss and origin failures for almost four hours
    Cloudflare2025
    Symptoms

    From 16:27 UTC on 21 August 2025, users connecting to Cloudflare from servers in AWS us-east-1, or receiving connections from Cloudflare there, saw high latency, packet loss and failures to reach origins. Cloudflare's network team was alerted to internal congestion in Ashburn (IAD) at 16:44 UTC.

    Packet lossHigh latencyIntermittent timeouts
    Trigger

    At about 16:27 UTC a single customer began sending many requests from AWS us-east-1 to Cloudflare for objects in Cloudflare's cache, doubling the total traffic from Cloudflare to AWS.

    Root cause
    Capacity exhaustion
    Component
    Transit gateway or network interconnectBGP routingCDN and edge network
    Contributing factor
    No headroomHarmful manual actionExcessive blast radius
    Detection

    Cloudflare's network team was alerted to internal congestion in Ashburn (IAD) at 16:44 UTC, and an incident was raised for customer impact at 17:45 UTC.

    Remediation
    • Rate limited the single customer whose traffic caused the surge
    • Took further traffic engineering actions that fully resolved the congestion
    • Asked AWS to revert its BGP prefix withdrawals, which AWS finished at 20:07 UTC
    Action items
    • Build a mechanism to deprioritise a customer's traffic when it congests the network enough to affect others
    • Speed up the Data Center Interconnect upgrades to add network capacity
    • Work with AWS so the two companies' BGP traffic engineering actions do not conflict
    • Build a traffic management system that gives each customer a network resource budget and automates the manual actions used here
    Lessons

    Cloudflare concludes that it must isolate customers better so that no single customer can take over shared network resources and degrade the platform for others, and that its network needs enough capacity to meet demand. It describes the outage as the result of insufficient network congestion management during an unusual high-traffic event.

    View case
  2. IL-0361 Job searches on seek.com.au failed for 16 minutes as a Transit Gateway silently dropped 9001-byte packets
    SEEK2021
    Symptoms

    In May 2021 some users of seek.com.au got errors when they searched for jobs, for 16 minutes. SEEK was in the middle of moving the traffic between two VPCs, the one that carried searches, from a VPC peering connection to an AWS Transit Gateway, and the team noticed quickly that search was broken.

    High error ratePacket lossIntermittent timeouts
    Trigger

    In the second step of the migration, SEEK changed the routing table of the first VPC to use the transit gateway while the second VPC still routed over the peering connection, so the two directions of each connection took different paths.

    Root cause
    Configuration change
    Component
    Transit gateway or network interconnect
    Contributing factor
    Unknown limitUnreviewed defaultsMonitoring blind spot
    Detection

    The team noticed soon after the routing change that search on seek.com.au was not working.

    Remediation
    • Rolled back the migration of the search traffic's peering connection to the transit gateway
    Action items
    • Lower the MTU of EC2 instances to 8500, on Amazon Linux 2 with a script at /sbin/ifup-local that runs whenever a network interface comes up
    Lessons

    A transit gateway has a smaller MTU than a same-region peering connection, drops oversized packets silently and clamps the MSS only on packets that cross it. When a migration leaves the two directions of a TCP connection on different paths, the clamp lands on the wrong side of the handshake and large packets are lost; the fix is not to send packets larger than 8500 bytes.

    View case
Check your cluster for this

These 2 cases involve Transit gateway or network interconnect and BGP routing. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause2

Capacity exhaustion1 case

Component
BGP routingCDN and edge networkTransit gateway or network interconnect
Contributing factor
Excessive blast radiusHarmful manual actionNo headroom
Organisation
Cloudflare
  1. IL-0341 Traffic between Cloudflare and AWS us-east-1 had high latency, packet loss and origin failures for almost four hours
    Cloudflare2025

Configuration change1 case

Component
Transit gateway or network interconnect
Contributing factor
Unreviewed defaultsUnknown limitMonitoring blind spot
Organisation
SEEK
  1. IL-0361 Job searches on seek.com.au failed for 16 minutes as a Transit Gateway silently dropped 9001-byte packets
    SEEK2021