Cases16 cases

9 to 13 of 16

1. IL-0267 Wikimedia's eqsin cache site went unreachable when both of its transport links to the core data centers were down

   Wikimedia Foundation2023

   Symptoms

   On 10 January 2023 at 16:37 UTC Icinga reported "BGP status on cr3-eqsin is CRITICAL: BGP CRITICAL - AS1299/IPv6: Active - Telia, AS1299/IPv4: Active - Telia", and from 16:45 the hosts in Wikimedia's eqsin site went DOWN with "PING CRITICAL - Packet loss = 100%". Users normally directed to eqsin, mostly in the APAC region, could only read Wikipedia pages already cached in eqsin.

   Network unreachable

   Trigger

   Planned maintenance by the transport provider took down eqsin's second transport link while the first was still down from a long fiber cut.

   Root cause

   Dependency failure

   Component

   Backbone network (WAN)BGP routingCDN and edge network

   Contributing factor

   Single point of failureIgnored early signal

   Detection

   Automated monitoring: Icinga BGP status alerts for cr3-eqsin at 16:37 and cr2-eqsin at 16:44, followed by a flood of host-down alerts and three pages at the same time.

   Remediation

   • Depooled eqsin from DNS
   • Repooled eqsin

   Action items

   • Create a backup GRE tunnel for eqsin (T327265)
   • Ensure the long-down transport link comes back up properly once it is fixed (T322529)
   • Automatically parse maintenance notifications and alert on conflicting maintenance (T230835)

   Lessons

   The site was depooled quickly, and "depool first, investigate later" was the right attitude. A flood of host-down alerts usually means a network problem. eqsin stayed too long on a single working transport link, SREs did not catch that the downtime of the two transport links overlapped, and the provider's planned maintenance was not recorded in a task or reminder.

   View case
2. IL-0266 Eqiad routers looped traffic between each other after a BGP MED change and could not reach other sites' public IPs

   Wikimedia Foundation2021

   Symptoms

   On 23 November 2021 at 09:39, two minutes after a change on the eqiad core routers, alerts fired because Icinga in eqiad could not reach the public text-lb address in eqsin (Singapore). For about 12 minutes eqiad could not reach hosts in other data centers, such as the cache PoPs, through their public IP addresses.

   Network unreachable

   Trigger

   At about 09:37 a change on cr1-eqiad and cr2-eqiad, made as part of task T295672 to influence BGP route selection, removed a BGP setting that sets the BGP "MED" attribute to the OSPF cost of reaching the BGP route's next hop.

   Root cause

   Configuration change

   Component

   BGP routing

   Contributing factor

   Ineffective rollback

   Detection

   Icinga health checks against edge caching clusters alerted at 09:39; the first alert was Icinga in eqiad failing to reach the public text-lb address in eqsin.

   Remediation

   • Reverted the configuration change on both eqiad routers
   • Forced a reset of the BGP session between the two eqiad routers

   Action items

   • Updated T295672 with further detail on the incident and a proposal to adjust the BGP sessions to address the "second IXP port" issue and prevent this from happening again

   View case
3. IL-0539 KT's internet service failed nationwide for about 89 minutes as wrong routes spread from one router in Busan

   KT2021

   Symptoms

   On 25 October 2021 from about 11:16, traffic to the DNS servers in KT's network rose sharply, and then KT's internet service failed across the whole country. KT recognised the internet outage at 11:20, and the outage lasted about 89 minutes until KT finished recovery at about 12:45.

   Network unreachableService outage

   Trigger

   During the replacement of an enterprise network router at KT's Busan office, worked on remotely from the Busan communication center, a worker entered a routing configuration script that lacked the 'exit' command that should have ended the IS-IS part of the configuration.

   Root cause

   Configuration change

   Component

   Interior routing (IGP)BGP routing

   Contributing factor

   Missed in reviewNo configuration testsChange during peak hoursExcessive blast radius

   Detection

   KT recognised the internet outage at 11:20, four minutes after DNS traffic began to surge at 11:16. The Ministry of Science and ICT saw the traffic increase in its own monitoring, and KT reported the outage to the ministry at 11:40.

   Remediation

   • Isolated the router where the faulty routing configuration was entered
   • Logged in to each regional router, deleted the faulty processes and restarted them

   Action items

   • Check the network management of major carriers, including their work procedures and the systems that stop technical errors from spreading
   • Consider having major carriers introduce simulation systems that diagnose errors from network work in advance
   • Have the NMS center and network operations center of each major carrier check technically that approved work plans are followed
   • Strengthen safeguards in routing work, such as limiting the number of route entries updated at once below a set level
   • Strengthen the monitoring of carriers' outages and develop technology, including AI-based preventive diagnosis, for network stability and resilience
   • Prepare structural measures for a stable network architecture and network survivability
   • Let carriers notify users of outages through SNS, text messages and other means, not only the website

   Lessons

   The investigation named management problems and technical problems. On the management side, work approved for the night was done during the day, partner company workers did the routing work without a work manager, and the work was done with the network still connected. On the technical side, the two pre-verification steps were people reading the script and missed the missing 'exit', there was no virtual testbed for trying the work on a disconnected network, and nothing stopped an error in one region from spreading to the whole country.

   View case
4. IL-0228 IPv4 routing across OVHcloud's global network broke after a router change put the full internet routing table into OSPF

   OVHcloud2021

   Symptoms

   At 09:20 CET on 13 October 2021, during a planned change on a router, IPv4 traffic across OVHcloud's whole network stopped being routed, while all IPv6 traffic stayed reachable. The team detected a problem with the router's performance at 09:21 CET and escalated at once.

   Network unreachableService outage

   Trigger

   A planned change to a router in the Vint Hill datacentre in the United States, meant to strengthen anti-DDoS protection, included a command to regulate the redistribution of BGP into OSPF, and the router did not interpret that command correctly.

   Root cause

   Configuration change

   Component

   Interior routing (IGP)BGP routingBackbone network (WAN)

   Contributing factor

   Ineffective rollbackExcessive blast radius

   Detection

   At 09:21 CET the team detected a problem with the router's performance and started the escalation process.

   Remediation

   • Disconnected the optical equipment to isolate the router's connectivity
   • Powered off the faulty router
   • Stabilised connectivity to restore the remaining services

   Action items

   • Re-evaluate the change procedure for this type of equipment, which applies the command line natively
   • Strengthen the related change process

   View case
5. IL-0221 A backbone maintenance command disconnected all Facebook data centers, and its DNS servers withdrew their BGP routes

   Meta2021

   Symptoms

   On 4 October 2021 Facebook's platforms went down worldwide: every connection in Facebook's global backbone network was lost, and the data centers were cut off from each other and from the internet. The rest of the internet could no longer find Facebook's servers, because Facebook's authoritative DNS servers had become unreachable.

   Service outageDNS resolution failuresNetwork unreachable

   Trigger

   During routine maintenance of the backbone, a command was issued to assess the availability of global backbone capacity, and it unintentionally took down all connections in the backbone network.

   Root cause

   Operator error

   Component

   Backbone network (WAN)BGP routingAuthoritative DNS servers and records

   Contributing factor

   Automation amplificationCircular dependencyLimited debug access

   Remediation

   • Sent engineers onsite to the data centers to debug the issue and restart systems
   • Brought the backbone network back online across the data center regions
   • Brought services back gradually, managing the rising load to avoid crashes from a traffic surge

   Action items

   • Find ways to simulate the global backbone being taken offline in storm drills
   • Strengthen testing, drills and overall resilience

   Lessons

   The hardening that protects Facebook's data centers against unauthorized access slowed recovery from an outage caused by Facebook's own error, and the author judges that tradeoff worth keeping. Storm drills prepared the teams to bring load back safely, although no drill had yet simulated the global backbone going offline.

   View case

By root cause6