Incident Library
Applied filters Root causeCapacity exhaustionComponentCDN and edge networkClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases3 cases

1 to 3 of 3

  1. IL-0353 Fly.io edge proxies in US west coast regions were overloaded and flapping, with lazy loader delays of seconds
    Fly.io2026
    Symptoms

    On 28 May 2026, Fly.io's edges in US west coast regions began flapping, first mainly in SJC. Logs and metrics of fly-proxy, the userspace L4 and L7 load balancer on the edges, showed lazy loader latency of 500 ms to several seconds, so many new requests waited that long or longer, while proxy CPU usage and the inbound connection rate were not especially high.

    High latency
    Root cause
    Capacity exhaustion
    Component
    CDN and edge networkService discovery
    Contributing factor
    No capacity planningUnknown limitOverlapping failures
    Detection

    Fly.io noticed flapping in its US west coast regions, and proxy logs and metrics showed high lazy loader latency.

    Remediation
    • Deployed the proxy in SJC with a temporary mitigation for one app with very large state in Corrosion, which helped only briefly
    • Shifted Anycast traffic from SJC to LAX
    • Lowered Airtime's bandwidth limit, which reduced softirq CPU usage and host load average
    • Bounced the proxy and hard-killed all proxy processes pending shutdown, which cleared the lazy loader problem
    Action items
    • Provision significantly more edge capacity
    • Fixed the bug that let multiple proxy instances coexist
    • Changed Airtime so that, for now, it applies a much stricter limit when it is not the expected active proxy instance
    • Fixed load shedding in the lazy loader so request latency has a reasonable upper bound
    • Shard the in-memory service catalog and move to finer-grained locking, tested with Antithesis
    • Rework Airtime to react to overall system load instead of only the proxy
    • Look into better monitoring for when the proxy is not in its expected configuration
    View case
  2. IL-0341 Traffic between Cloudflare and AWS us-east-1 had high latency, packet loss and origin failures for almost four hours
    Cloudflare2025
    Symptoms

    From 16:27 UTC on 21 August 2025, users connecting to Cloudflare from servers in AWS us-east-1, or receiving connections from Cloudflare there, saw high latency, packet loss and failures to reach origins. Cloudflare's network team was alerted to internal congestion in Ashburn (IAD) at 16:44 UTC.

    Packet lossHigh latencyIntermittent timeouts
    Trigger

    At about 16:27 UTC a single customer began sending many requests from AWS us-east-1 to Cloudflare for objects in Cloudflare's cache, doubling the total traffic from Cloudflare to AWS.

    Root cause
    Capacity exhaustion
    Component
    Transit gateway or network interconnectBGP routingCDN and edge network
    Contributing factor
    No headroomHarmful manual actionExcessive blast radius
    Detection

    Cloudflare's network team was alerted to internal congestion in Ashburn (IAD) at 16:44 UTC, and an incident was raised for customer impact at 17:45 UTC.

    Remediation
    • Rate limited the single customer whose traffic caused the surge
    • Took further traffic engineering actions that fully resolved the congestion
    • Asked AWS to revert its BGP prefix withdrawals, which AWS finished at 20:07 UTC
    Action items
    • Build a mechanism to deprioritise a customer's traffic when it congests the network enough to affect others
    • Speed up the Data Center Interconnect upgrades to add network capacity
    • Work with AWS so the two companies' BGP traffic engineering actions do not conflict
    • Build a traffic management system that gives each customer a network resource budget and automates the manual actions used here
    Lessons

    Cloudflare concludes that it must isolate customers better so that no single customer can take over shared network resources and degrade the platform for others, and that its network needs enough capacity to meet demand. It describes the outage as the result of insufficient network congestion management during an unusual high-traffic event.

    View case
  3. IL-0555 Sentry's ingestion edge regions backlogged one after another and rejected traffic in three incidents in October 2024
    Sentry2024
    Symptoms

    On 21 October 2024 Sentry could not ingest data in US SaaS from 9:42 AM to 10:06 AM PT, and the status page reported elevated errors for ingestion. The same failure had also happened on 4 and 16 October.

    High error rateService outageAutoscaler misbehaviour
    Trigger

    In each of the three incidents an ingestion region saw an increase in backlogged traffic; the trigger was different each time.

    Root cause
    Capacity exhaustion
    Component
    CDN and edge networkAutomated capacity manager
    Contributing factor
    Automation amplificationExcessive blast radius
    Action items
    • Create more, smaller edge regions to reduce the chance of cascading failures, adding one region each in South America, North America and Europe
    • Change the autoscaling rules to use stronger signals than CPU load
    • Scale down more slowly so that iowait-based backlogs do not trigger scale-down events
    • Improve the metrics the edge produces to refine the autoscaling rules and their sensitivity
    View case
Check your cluster for this

These 3 cases involve CDN and edge network and BGP routing. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Capacity exhaustion3 cases

Component
CDN and edge network3BGP routing1Automated capacity manager1Transit gateway or network interconnect1Service discovery1
Contributing factor
Excessive blast radius2Automation amplification1Unknown limit1Harmful manual action1No capacity planning1No headroom1Overlapping failures1
Organisation
Cloudflare1Fly.io1Sentry1
  1. IL-0353 Fly.io edge proxies in US west coast regions were overloaded and flapping, with lazy loader delays of seconds
    Fly.io2026
  2. IL-0341 Traffic between Cloudflare and AWS us-east-1 had high latency, packet loss and origin failures for almost four hours
    Cloudflare2025
  3. IL-0555 Sentry's ingestion edge regions backlogged one after another and rejected traffic in three incidents in October 2024
    Sentry2024