Cases7 cases

1 to 5 of 7

1. IL-0344 Linode's US-West data centers in Fremont and Santa Clara lost connectivity as the fibre circuits between them failed

   Linode2023

   Symptoms

   On 12 February 2023, customers in Linode's US-West region could not reach their Linodes. Santa Clara customers lost Internet connectivity after a change made at 8:45 UTC until it was reverted at 10:13 UTC, and later in the day the connection between the Fremont and Santa Clara data centers was cut completely, leaving both with degraded service or a full outage until 16:25 UTC; customers could also have trouble provisioning, powering down or rebooting services.

   Network unreachable

   Trigger

   A single link of the A-side fibre circuit between Fremont and Santa Clara went down on 11 February, the whole B-side circuit followed on 12 February, and during troubleshooting with the hardware vendor a non-intrusive command triggered a rare bug that took down all A-side links.

   Root cause

   Software defect

   Component

   Backbone network (WAN)NIC, cable and switch port link

   Contributing factor

   Overlapping failuresHarmful manual action

   Detection

   The response team began investigating when the A-side link went down on 11 February. The status page opened at 10:22 UTC on 12 February to investigate reports of connectivity issues in the US-West region.

   Remediation

   • Reverted the change that removed internet traffic from the Santa Clara border routers
   • Performed an emergency code upgrade of both circuits, restoring connectivity between the sites
   • Restored the B-side path with the hardware vendor and fiber provider, regaining full redundancy

   View case
2. IL-0225 An IGP purge command made every Microsoft WAN router recompute routes, causing global latency, timeouts and packet loss

   Microsoft Azure2023

   Symptoms

   From 07:08 UTC on 25 January 2023, customers saw long network latency and timeouts when connecting to resources in Azure regions and to other Microsoft services such as Microsoft 365 and Power Platform. Microsoft's monitoring detected DNS and WAN issues from 07:11 UTC.

   High latencyIntermittent timeoutsIntermittent connection failuresPacket lossNetwork unreachable

   Trigger

   A network engineer adding capacity to the global WAN in Madrid ran a step of a recently changed standard operating procedure that purged the IGP database; on the router manufacturer involved, that command runs on all IGP-joined routers instead of only the local one.

   Root cause

   Configuration change

   Component

   Backbone network (WAN)Interior routing (IGP)BGP routing

   Contributing factor

   No configuration testsUnverified assumptionExcessive blast radiusHarmful manual action

   Detection

   Monitoring detected DNS and WAN issues from 07:11 UTC and alerted the on-call engineers within 5 minutes of the command being run. The team reviewed all recent changes and identified the command by 08:20 UTC. The engineer making the change was not informed of the alerts because of the unqualified changes to the SOP.

   Remediation

   • Restarted by hand the paused WAN health and traffic engineering systems, which ended the remaining packet loss

   Action items

   • Audit and block similar widespread-impact commands across all three vendors for all WAN router roles
   • Give on-call engineers real-time visibility of approved and unqualified activity on network devices
   • Run regular mandatory operational training and attestation of following all SOPs
   • Have the Change Advisory Board review all SOPs still pending qualification within 30 days

   View case
3. IL-0267 Wikimedia's eqsin cache site went unreachable when both of its transport links to the core data centers were down

   Wikimedia Foundation2023

   Symptoms

   On 10 January 2023 at 16:37 UTC Icinga reported "BGP status on cr3-eqsin is CRITICAL: BGP CRITICAL - AS1299/IPv6: Active - Telia, AS1299/IPv4: Active - Telia", and from 16:45 the hosts in Wikimedia's eqsin site went DOWN with "PING CRITICAL - Packet loss = 100%". Users normally directed to eqsin, mostly in the APAC region, could only read Wikipedia pages already cached in eqsin.

   Network unreachable

   Trigger

   Planned maintenance by the transport provider took down eqsin's second transport link while the first was still down from a long fiber cut.

   Root cause

   Dependency failure

   Component

   Backbone network (WAN)BGP routingCDN and edge network

   Contributing factor

   Single point of failureIgnored early signal

   Detection

   Automated monitoring: Icinga BGP status alerts for cr3-eqsin at 16:37 and cr2-eqsin at 16:44, followed by a flood of host-down alerts and three pages at the same time.

   Remediation

   • Depooled eqsin from DNS
   • Repooled eqsin

   Action items

   • Create a backup GRE tunnel for eqsin (T327265)
   • Ensure the long-down transport link comes back up properly once it is fixed (T322529)
   • Automatically parse maintenance notifications and alert on conflicting maintenance (T230835)

   Lessons

   The site was depooled quickly, and "depool first, investigate later" was the right attitude. A flood of host-down alerts usually means a network problem. eqsin stayed too long on a single working transport link, SREs did not catch that the downtime of the two transport links overlapped, and the provider's planned maintenance was not recorded in a task or reminder.

   View case
4. IL-0228 IPv4 routing across OVHcloud's global network broke after a router change put the full internet routing table into OSPF

   OVHcloud2021

   Symptoms

   At 09:20 CET on 13 October 2021, during a planned change on a router, IPv4 traffic across OVHcloud's whole network stopped being routed, while all IPv6 traffic stayed reachable. The team detected a problem with the router's performance at 09:21 CET and escalated at once.

   Network unreachableService outage

   Trigger

   A planned change to a router in the Vint Hill datacentre in the United States, meant to strengthen anti-DDoS protection, included a command to regulate the redistribution of BGP into OSPF, and the router did not interpret that command correctly.

   Root cause

   Configuration change

   Component

   Interior routing (IGP)BGP routingBackbone network (WAN)

   Contributing factor

   Ineffective rollbackExcessive blast radius

   Detection

   At 09:21 CET the team detected a problem with the router's performance and started the escalation process.

   Remediation

   • Disconnected the optical equipment to isolate the router's connectivity
   • Powered off the faulty router
   • Stabilised connectivity to restore the remaining services

   Action items

   • Re-evaluate the change procedure for this type of equipment, which applies the command line natively
   • Strengthen the related change process

   View case
5. IL-0221 A backbone maintenance command disconnected all Facebook data centers, and its DNS servers withdrew their BGP routes

   Meta2021

   Symptoms

   On 4 October 2021 Facebook's platforms went down worldwide: every connection in Facebook's global backbone network was lost, and the data centers were cut off from each other and from the internet. The rest of the internet could no longer find Facebook's servers, because Facebook's authoritative DNS servers had become unreachable.

   Service outageDNS resolution failuresNetwork unreachable

   Trigger

   During routine maintenance of the backbone, a command was issued to assess the availability of global backbone capacity, and it unintentionally took down all connections in the backbone network.

   Root cause

   Operator error

   Component

   Backbone network (WAN)BGP routingAuthoritative DNS servers and records

   Contributing factor

   Automation amplificationCircular dependencyLimited debug access

   Remediation

   • Sent engineers onsite to the data centers to debug the issue and restart systems
   • Brought the backbone network back online across the data center regions
   • Brought services back gradually, managing the rising load to avoid crashes from a traffic surge

   Action items

   • Find ways to simulate the global backbone being taken offline in storm drills
   • Strengthen testing, drills and overall resilience

   Lessons

   The hardening that protects Facebook's data centers against unauthorized access slowed recovery from an outage caused by Facebook's own error, and the author judges that tradeoff worth keeping. Storm drills prepared the teams to bring load back safely, although no drill had yet simulated the global backbone going offline.

   View case

By root cause4