IL-0341 Cloudflare · 2025-08-21networkingaws

Traffic between Cloudflare and AWS us-east-1 had high latency, packet loss and origin failures for almost four hours

Most Cloudflare customers run their own origin servers and put Cloudflare in front of them as a reverse proxy that serves cached content where it can. Responses from cache travel over Cloudflare's internal Data Center Interconnect (DCI) links through a series of network equipment to its edge routers. On 21 August 2025 a single customer in AWS us-east-1 began requesting large amounts of cached content at 16:27 UTC, and the response traffic saturated all direct peering connections between Cloudflare and AWS. One of the direct peering links was already running at half capacity because of an earlier failure, and the DCI that connects Cloudflare's edge routers to an offsite interconnection switch was due for a capacity upgrade.

To ease the congestion, AWS withdrew some BGP advertisements to Cloudflare over the congested links from 16:37 UTC, which pushed traffic onto further peering links through the offsite switch; those links saturated as well and from 17:22 UTC more traffic was dropped. Cloudflare raised an incident for customer impact in Ashburn (IAD) at 17:45 UTC and worked with AWS on manual traffic actions. Rate limiting the customer that caused the surge reduced the congestion at 19:05 UTC, and further traffic engineering by the network team ended it at 19:27 UTC. The impact was substantially reduced by 19:38 UTC, with intermittent latency increases continuing after that. At Cloudflare's request AWS began reverting the BGP withdrawals at 19:45 UTC and finished restoring the prefix announcements over the IAD PNIs at 20:07 UTC, and the impact ended at 20:18 UTC. Cloudflare states the event was network congestion, not an attack or a BGP hijack.

Severity: –
Scope: traffic between Cloudflare and AWS us-east-1, including requests to customer origins hosted there
Users affected: customers with origins in AWS us-east-1 and users connecting to or from Cloudflare via servers in AWS us-east-1
Region: AWS us-east-1 and Cloudflare's Ashburn (IAD) location
Time to detect: 17m
Time to resolve: 3h 51m

Symptoms

From 16:27 UTC on 21 August 2025, users connecting to Cloudflare from servers in AWS us-east-1, or receiving connections from Cloudflare there, saw high latency, packet loss and failures to reach origins. Cloudflare's network team was alerted to internal congestion in Ashburn (IAD) at 16:44 UTC.

The links between Cloudflare and AWS us-east-1 were saturated, router queues grew and Cloudflare's Ashburn edge routers dropped high-priority packets for the whole period. For customers this showed as latency, loss that ended in timeouts, or low throughput. Cloudflare's latency Service Level Objectives, which imitate customer requests back to their origins, showed the share of requests within the latency target falling below an acceptable level in step with the packet drops. Global Cloudflare services and traffic outside the path to AWS us-east-1 were not affected.

The congestion got worse after it began. At 16:37 UTC AWS started withdrawing prefixes from Cloudflare on the congested Private Network Interconnect (PNI) BGP sessions to steer traffic elsewhere; the withdrawals moved traffic to other peering links behind an offsite network interconnection switch, which then saturated too, and from 17:22 UTC more traffic was dropped. Cloudflare's network team could not simply move traffic to uncongested paths, because the withdrawn AWS prefixes were no longer available there. After the congestion ended, both sides spent time putting the prefix announcements back, so customer latency recovered later than the packet drops did.

Symptom Packet loss High latency Intermittent timeouts

Root cause

Root cause Capacity exhaustion

Component Transit gateway or network interconnect BGP routing CDN and edge network

Contributing factor No headroom Harmful manual action Excessive blast radius

Trigger: At about 16:27 UTC a single customer began sending many requests from AWS us-east-1 to Cloudflare for objects in Cloudflare's cache, doubling the total traffic from Cloudflare to AWS.

Detection

Cloudflare's network team was alerted to internal congestion in Ashburn (IAD) at 16:44 UTC, and an incident was raised for customer impact at 17:45 UTC.

Remediation

Rate limited the single customer whose traffic caused the surge
Took further traffic engineering actions that fully resolved the congestion
Asked AWS to revert its BGP prefix withdrawals, which AWS finished at 20:07 UTC

Action items

Build a mechanism to deprioritise a customer's traffic when it congests the network enough to affect others
Speed up the Data Center Interconnect upgrades to add network capacity
Work with AWS so the two companies' BGP traffic engineering actions do not conflict
Build a traffic management system that gives each customer a network resource budget and automates the manual actions used here

Lessons

Cloudflare concludes that it must isolate customers better so that no single customer can take over shared network resources and degrade the platform for others, and that its network needs enough capacity to meet demand. It describes the outage as the result of insufficient network congestion management during an unusual high-traffic event.

Sources

https://blog.cloudflare.com/cloudflare-incident-on-august-21-2025/ · blog

Related cases

IL-0353 Fly.io edge proxies in US west coast regions were overloaded and flapping, with lazy loader delays of secondsFly.io · 2026
same root cause · shared CDN and edge network
IL-0350 Hosts in an IONOS TXL cluster lost connectivity when an InfiniBand fabric resweep failed on a full multicast tableIONOS Cloud · 2026
same root cause · shared BGP routing
IL-0564 Linode's Miami, Milan and Sao Paulo data centers had packet loss as traffic moved onto congested alternate pathsLinode · 2025
same root cause · shared BGP routing
IL-0555 Sentry's ingestion edge regions backlogged one after another and rejected traffic in three incidents in October 2024Sentry · 2024
same root cause · shared CDN and edge network

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Transit gateway or network interconnect, BGP routing, CDN and edge network among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0341
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0341/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.