Cases7 cases

1 to 5 of 7

1. IL-0341 Traffic between Cloudflare and AWS us-east-1 had high latency, packet loss and origin failures for almost four hours

   Cloudflare2025

   Symptoms

   From 16:27 UTC on 21 August 2025, users connecting to Cloudflare from servers in AWS us-east-1, or receiving connections from Cloudflare there, saw high latency, packet loss and failures to reach origins. Cloudflare's network team was alerted to internal congestion in Ashburn (IAD) at 16:44 UTC.

   Packet lossHigh latencyIntermittent timeouts

   Trigger

   At about 16:27 UTC a single customer began sending many requests from AWS us-east-1 to Cloudflare for objects in Cloudflare's cache, doubling the total traffic from Cloudflare to AWS.

   Root cause

   Capacity exhaustion

   Component

   Transit gateway or network interconnectBGP routingCDN and edge network

   Contributing factor

   No headroomHarmful manual actionExcessive blast radius

   Detection

   Cloudflare's network team was alerted to internal congestion in Ashburn (IAD) at 16:44 UTC, and an incident was raised for customer impact at 17:45 UTC.

   Remediation

   • Rate limited the single customer whose traffic caused the surge
   • Took further traffic engineering actions that fully resolved the congestion
   • Asked AWS to revert its BGP prefix withdrawals, which AWS finished at 20:07 UTC

   Action items

   • Build a mechanism to deprioritise a customer's traffic when it congests the network enough to affect others
   • Speed up the Data Center Interconnect upgrades to add network capacity
   • Work with AWS so the two companies' BGP traffic engineering actions do not conflict
   • Build a traffic management system that gives each customer a network resource budget and automates the manual actions used here

   Lessons

   Cloudflare concludes that it must isolate customers better so that no single customer can take over shared network resources and degrade the platform for others, and that its network needs enough capacity to meet demand. It describes the outage as the result of insufficient network congestion management during an unusual high-traffic event.

   View case
2. IL-0347 About 25% of Scaleway FR-PAR-1 hypervisors lost VPC connectivity when both VPC BGP route reflectors got stuck

   Scaleway2024

   Symptoms

   On 18 October 2024, from 7:50 to 8:01 UTC, about 25% of the hypervisors in Scaleway's FR-PAR-1 availability zone lost VPC connectivity with the rest of the region, and products that use VPC failed with them. Public internet connectivity and Elastic Metal VPC connectivity were not affected.

   Network unreachablePacket lossNode NotReady

   Trigger

   At 6:20 UTC a Top of Rack switch in FR-PAR-1 had a software crash. One of the two BGP route reflectors for VPC ran on a hypervisor in that rack and was left in an abnormal state.

   Root cause

   Software defect

   Component

   Network control plane (SDN)BGP routingNetwork switchDatabaseManaged Kubernetes control plane and provider API

   Contributing factor

   Excessive blast radiusAutomation amplification

   Detection

   Health-check monitoring on both route reflectors detected the anomaly, and the Scaleway Customer Excellence and SRE teams detected the customer impact and opened an incident.

   Remediation

   • Fixed both VPC route reflectors by manual action

   Action items

   • Fixed the autohealing mechanism that had failed to fix route reflectors stuck in a stale state; the fix is https://github.com/FRRouting/frr/pull/17163
   • Run different software versions on the route reflectors so a single bug cannot stop several instances
   • Investigate the software issue that left the route reflectors in a stale state

   View case
3. IL-0339 Azure Front Door and Azure CDN users saw intermittent connection errors, timeouts and latency spikes for hours

   Microsoft Azure2024

   Symptoms

   From 11:45 UTC on 30 July 2024, a subset of customers got intermittent connection errors, timeouts and latency spikes when connecting to Microsoft services that use Azure Front Door (AFD) and Azure Content Delivery Network (CDN). The Azure Portal team saw the first service degradation at 11:47 UTC.

   Intermittent connection failuresIntermittent timeoutsHigh latencyPacket loss

   Trigger

   Around 11:45 UTC the Azure network DDoS protection service was disengaging after a routine DDoS mitigation and returning traffic to its default routing to AFD, but the routes at one site in Europe could not be updated because the network control plane could not reach that site during a local power outage.

   Root cause

   Configuration change

   Component

   CDN and edge networkDDoS protection serviceNetwork control plane (SDN)Data center power and cooling

   Contributing factor

   Overlapping failuresExcessive blast radiusMonitoring blind spot

   Detection

   The Azure Portal team detected service degradation at 11:47 UTC and began to investigate. At 12:10 UTC network monitoring correlated the Portal incident to a network issue at one site in Europe, and networking engineers joined.

   Remediation

   • Rerouted traffic under the standard congestion playbook, which removed most of the impact by 13:58 UTC
   • Ordered the network control plane to reissue its commands to the misbehaving device, which could not be reached
   • Removed the device that kept attracting traffic from the network and scanned the network for other issues

   Action items

   • Add the missing configuration on network devices so a DDoS mitigation issue in one geography cannot spread to another
   • Improve validation and monitoring in the Azure network to detect invalid configurations
   • Improve monitoring for when the DDoS protection service is unreachable from the control plane but still serving traffic

   View case
4. IL-0529 New Google Cloud VMs and other instances came up without network connectivity in all zones after VPC controllers stopped

   Google Cloud2024

   Symptoms

   From 15:22 US/Pacific on 16 May 2024, new VMs and other new instances across Google Cloud came up without network programming, so the new instances could not open network connections, and VMs that were migrated or restarted lost connectivity. Changes to virtual network configuration, such as firewall rules and load balancers, were not applied. Internal monitoring alerted Google engineers at 15:41.

   Network unreachablePacket lossCluster or node provisioning stallScale-up failureHigh error rate

   Trigger

   Google's infrastructure team ran a routine maintenance action meant to shut down one unused VPC Controller in a single Google Cloud zone, and the automation shut down VPC controllers in all zones where the controllers were still in use.

   Root cause

   Software defect

   Component

   Network control plane (SDN)Cloud load balancerNAT gateway

   Contributing factor

   Change with a delayed effectIgnored early signalNo safeguard on destructive changeExcessive blast radius

   Detection

   Internal monitoring alerts reached Google engineers at 15:41 US/Pacific, and the investigation started immediately.

   Remediation

   • Paused maintenance operations to prevent the shutdown from repeating
   • Brought the affected VPC Controllers back into a serving state in every zone
   • Stopped the automation responsible for the shutdown until safeguards are in place
   • Recovered partial Apigee proxy deployments by hand
   • Cleaned up VMs left behind by failed Dataproc cluster deletions

   Action items

   • Add safety measures that check that maintenance operations carry correctly scoped zone information
   • Make the underlying tools reject maintenance operations that do not specify zones
   • Tighten on-demand access control so one maintenance invocation cannot work across multiple zones
   • Add safety checks, testing and validation for maintenance operations, including positive and negative test cases for the new safety measures

   View case
5. IL-0530 Virtual router controllers in us-central1 ran out of memory in a loop, cutting networking for new and migrated VMs

   Google Cloud2023

   Symptoms

   On 5 October 2023 in us-central1, newly created VMs waited a long time before their networking worked, and VMs that had been live migrated lost connectivity for a long time after the migration. Networking telemetry showed cross-region packet loss to and from us-central1 from 03:08 US/Pacific. Health check state of VMs reached load balancers late.

   Packet lossNetwork unreachableOOMKilledStale endpoint routingCluster or node provisioning stall

   Trigger

   A management plane behaviour change, meant to decouple the processing of API updates to Instance Groups and Network Endpoint Groups used as load balancer backends, was deployed in us-central1 after rolling out to several other regions without incident.

   Root cause

   Faulty deployment

   Component

   Network control plane (SDN)Cloud load balancer

   Contributing factor

   Excessive blast radiusNo alert on limit

   Detection

   Engineers were alerted to slowness in the virtual network control plane in us-central1 at 21:45 US/Pacific on 4 October, before the packet loss started, and to controller failures from memory exhaustion at 02:11 on 5 October. Networking telemetry showed cross-region packet loss from 03:08.

   Remediation

   • Increased the memory allocated to the virtual network router controllers
   • Rolled back the management plane change in us-central1
   • Restarted by hand the routers that were still not accepting updates

   Action items

   • Alert proactively on memory risks and unexpected memory increases
   • Refactor the deployment configuration so engineers can reallocate memory much faster
   • Re-evaluate existing practices and add safety mechanisms for API reconciliation
   • Make management plane changes more visible across teams so the changes can be correlated faster
   • Adjust the deployment footprint so a regional API change is less likely to exhaust memory across the region at once
   • Optimise memory use in the traffic routers and their controllers

   View case

By root cause4