Cases4 cases

1 to 4 of 4

1. IL-0348 Scaleway's FR-PAR region lost public internet connectivity when one 100 Gbps backbone link was saturated

   Scaleway2025

   Symptoms

   On 10 March 2025 at 16:22 UTC, public internet connectivity in Scaleway's FR-PAR region broke for all products. Most internet traffic between the FR-PAR-2 availability zone and the rest of the PAR region was blackholed for about five minutes, while VPC traffic and Dedibox RPN kept working.

   Packet lossService outageHigh latency

   Trigger

   While Scaleway's network team connected a new data center to the FR-PAR backbone, a human mistake in a router configuration rerouted more than 1 Tbps of traffic between OpCore DC3 and DC5 through a single 100 Gbps link.

   Root cause

   Operator error

   Component

   Backbone network (WAN)Object storageAuthoritative DNS servers and records

   Contributing factor

   Automation amplification

   Detection

   The router configuration error was detected quickly and rolled back within four minutes.

   Remediation

   • Rolled back the router configuration, restoring the correct configuration at 16:27 UTC

   Action items

   • Harden the peer review and production communication process for changes
   • Add failsafes and tests before network configuration is applied at backbone level
   • Deploy the fix for the Object Storage synchronisation software issue

   View case
2. IL-0228 IPv4 routing across OVHcloud's global network broke after a router change put the full internet routing table into OSPF

   OVHcloud2021

   Symptoms

   At 09:20 CET on 13 October 2021, during a planned change on a router, IPv4 traffic across OVHcloud's whole network stopped being routed, while all IPv6 traffic stayed reachable. The team detected a problem with the router's performance at 09:21 CET and escalated at once.

   Network unreachableService outage

   Trigger

   A planned change to a router in the Vint Hill datacentre in the United States, meant to strengthen anti-DDoS protection, included a command to regulate the redistribution of BGP into OSPF, and the router did not interpret that command correctly.

   Root cause

   Configuration change

   Component

   Interior routing (IGP)BGP routingBackbone network (WAN)

   Contributing factor

   Ineffective rollbackExcessive blast radius

   Detection

   At 09:21 CET the team detected a problem with the router's performance and started the escalation process.

   Remediation

   • Disconnected the optical equipment to isolate the router's connectivity
   • Powered off the faulty router
   • Stabilised connectivity to restore the remaining services

   Action items

   • Re-evaluate the change procedure for this type of equipment, which applies the command line natively
   • Strengthen the related change process

   View case
3. IL-0221 A backbone maintenance command disconnected all Facebook data centers, and its DNS servers withdrew their BGP routes

   Meta2021

   Symptoms

   On 4 October 2021 Facebook's platforms went down worldwide: every connection in Facebook's global backbone network was lost, and the data centers were cut off from each other and from the internet. The rest of the internet could no longer find Facebook's servers, because Facebook's authoritative DNS servers had become unreachable.

   Service outageDNS resolution failuresNetwork unreachable

   Trigger

   During routine maintenance of the backbone, a command was issued to assess the availability of global backbone capacity, and it unintentionally took down all connections in the backbone network.

   Root cause

   Operator error

   Component

   Backbone network (WAN)BGP routingAuthoritative DNS servers and records

   Contributing factor

   Automation amplificationCircular dependencyLimited debug access

   Remediation

   • Sent engineers onsite to the data centers to debug the issue and restart systems
   • Brought the backbone network back online across the data center regions
   • Brought services back gradually, managing the rising load to avoid crashes from a traffic surge

   Action items

   • Find ways to simulate the global backbone being taken offline in storm drills
   • Strengthen testing, drills and overall resilience

   Lessons

   The hardening that protects Facebook's data centers against unauthorized access slowed recovery from an outage caused by Facebook's own error, and the author judges that tradeoff worth keeping. Storm drills prepared the teams to bring load back safely, although no drill had yet simulated the global backbone going offline.

   View case
4. IL-0256 Basecamp was fully down twice in two days as hardware load balancers crashed and routers lost sync after failover

   37signals2020

   Symptoms

   On Tuesday 1 September 2020 Basecamp could not be loaded at all for 17 minutes, from 9:51am to 10:08am Central Time (14:51–15:08 UTC), and nothing seemed to work. On Wednesday 2 September it happened again for 15 minutes, from 10:50am to 11:05am Central Time (15:50–16:05 UTC).

   Service outageLink flappingHigh latency

   Trigger

   On 1 September the network links between the data centers went offline, and during the recovery a hardware load balancer crashed. On 2 September the primary load balancer in Virginia crashed again, and later its secondary crashed too.

   Root cause

   Software defect

   Component

   On-premises load balancerBackbone network (WAN)

   Contributing factor

   Overlapping failuresNo headroom

   Remediation

   • On 1 September, intervened by hand to get the network routers working with the standby load balancer
   • On 2 September, fixed the same router and load balancer desynchronization by hand

   Action items

   • Find the cause of the load balancer crash with the vendor and apply its bugfix
   • Replace the hardware load balancers
   • Reroute the redundant cross-datacenter network paths for proper circuit diversity
   • Evaluate moving from hardware to software load balancers, which can be deployed in minutes
   • Evaluate decentralizing the load balancer architecture to limit the impact of one failure
   • Assess the systems for resilience, fragility and risk, and review the assessment process itself

   Lessons

   Basecamp plans for failures like these and practises responses to catastrophic failures in its live systems, but three obscure issues in five days chained into overlapping failures. What caught the team by surprise were cascading failures that exposed unexpected fragility and hard paths to recovery, which Basecamp says it can prepare for.

   View case

By root cause3