Cases3 cases

1 to 3 of 3

1. IL-0353 Fly.io edge proxies in US west coast regions were overloaded and flapping, with lazy loader delays of seconds

   Fly.io2026

   Symptoms

   On 28 May 2026, Fly.io's edges in US west coast regions began flapping, first mainly in SJC. Logs and metrics of fly-proxy, the userspace L4 and L7 load balancer on the edges, showed lazy loader latency of 500 ms to several seconds, so many new requests waited that long or longer, while proxy CPU usage and the inbound connection rate were not especially high.

   High latency

   Root cause

   Capacity exhaustion

   Component

   CDN and edge networkService discovery

   Contributing factor

   No capacity planningUnknown limitOverlapping failures

   Detection

   Fly.io noticed flapping in its US west coast regions, and proxy logs and metrics showed high lazy loader latency.

   Remediation

   • Deployed the proxy in SJC with a temporary mitigation for one app with very large state in Corrosion, which helped only briefly
   • Shifted Anycast traffic from SJC to LAX
   • Lowered Airtime's bandwidth limit, which reduced softirq CPU usage and host load average
   • Bounced the proxy and hard-killed all proxy processes pending shutdown, which cleared the lazy loader problem

   Action items

   • Provision significantly more edge capacity
   • Fixed the bug that let multiple proxy instances coexist
   • Changed Airtime so that, for now, it applies a much stricter limit when it is not the expected active proxy instance
   • Fixed load shedding in the lazy loader so request latency has a reasonable upper bound
   • Shard the in-memory service catalog and move to finer-grained locking, tested with Antithesis
   • Rework Airtime to react to overall system load instead of only the proxy
   • Look into better monitoring for when the proxy is not in its expected configuration

   View case
2. IL-0341 Traffic between Cloudflare and AWS us-east-1 had high latency, packet loss and origin failures for almost four hours

   Cloudflare2025

   Symptoms

   From 16:27 UTC on 21 August 2025, users connecting to Cloudflare from servers in AWS us-east-1, or receiving connections from Cloudflare there, saw high latency, packet loss and failures to reach origins. Cloudflare's network team was alerted to internal congestion in Ashburn (IAD) at 16:44 UTC.

   Packet lossHigh latencyIntermittent timeouts

   Trigger

   At about 16:27 UTC a single customer began sending many requests from AWS us-east-1 to Cloudflare for objects in Cloudflare's cache, doubling the total traffic from Cloudflare to AWS.

   Root cause

   Capacity exhaustion

   Component

   Transit gateway or network interconnectBGP routingCDN and edge network

   Contributing factor

   No headroomHarmful manual actionExcessive blast radius

   Detection

   Cloudflare's network team was alerted to internal congestion in Ashburn (IAD) at 16:44 UTC, and an incident was raised for customer impact at 17:45 UTC.

   Remediation

   • Rate limited the single customer whose traffic caused the surge
   • Took further traffic engineering actions that fully resolved the congestion
   • Asked AWS to revert its BGP prefix withdrawals, which AWS finished at 20:07 UTC

   Action items

   • Build a mechanism to deprioritise a customer's traffic when it congests the network enough to affect others
   • Speed up the Data Center Interconnect upgrades to add network capacity
   • Work with AWS so the two companies' BGP traffic engineering actions do not conflict
   • Build a traffic management system that gives each customer a network resource budget and automates the manual actions used here

   Lessons

   Cloudflare concludes that it must isolate customers better so that no single customer can take over shared network resources and degrade the platform for others, and that its network needs enough capacity to meet demand. It describes the outage as the result of insufficient network congestion management during an unusual high-traffic event.

   View case
3. IL-0555 Sentry's ingestion edge regions backlogged one after another and rejected traffic in three incidents in October 2024

   Sentry2024

   Symptoms

   On 21 October 2024 Sentry could not ingest data in US SaaS from 9:42 AM to 10:06 AM PT, and the status page reported elevated errors for ingestion. The same failure had also happened on 4 and 16 October.

   High error rateService outageAutoscaler misbehaviour

   Trigger

   In each of the three incidents an ingestion region saw an increase in backlogged traffic; the trigger was different each time.

   Root cause

   Capacity exhaustion

   Component

   CDN and edge networkAutomated capacity manager

   Contributing factor

   Automation amplificationExcessive blast radius

   Action items

   • Create more, smaller edge regions to reduce the chance of cascading failures, adding one region each in South America, North America and Europe
   • Change the autoscaling rules to use stronger signals than CPU load
   • Scale down more slowly so that iowait-based backlogs do not trigger scale-down events
   • Improve the metrics the edge produces to refine the autoscaling rules and their sensitivity

   View case

By root cause1