Cases6 cases

3 to 6 of 6

1. IL-0259 Outbound packets from Fastmail did not reach part of the internet for about 19 hours, cutting off 3-5% of customers

   Fastmail2023

   Symptoms

   From about 03:00 UTC on 30 June 2023, Fastmail and a fraction of the internet could not communicate with each other, and Fastmail first heard of it from customers through support tickets and social media. The problem lasted about 19 hours and affected an estimated 3-5% of customers as well as some email servers.

   Network unreachable

   Root cause

   Dependency failure

   Component

   ISP and transit networks

   Contributing factor

   Single point of failureMonitoring blind spot

   Detection

   Customers told Fastmail through support tickets and social media. Fastmail's own monitors in multiple locations and its staff were not on affected networks and saw nothing.

   Action items

   • Select a provider for a second transit connection directly into Fastmail's servers, via Megaport or a provider present in 365's New Jersey datacenter, so that Fastmail controls the outbound path and can route around a network with problems

   Lessons

   With a single path for outbound traffic, Fastmail could not route around a provider's network problem and could not fix it itself. Being able to choose between several outbound paths would let Fastmail get everyone back online faster.

   View case
2. IL-0267 Wikimedia's eqsin cache site went unreachable when both of its transport links to the core data centers were down

   Wikimedia Foundation2023

   Symptoms

   On 10 January 2023 at 16:37 UTC Icinga reported "BGP status on cr3-eqsin is CRITICAL: BGP CRITICAL - AS1299/IPv6: Active - Telia, AS1299/IPv4: Active - Telia", and from 16:45 the hosts in Wikimedia's eqsin site went DOWN with "PING CRITICAL - Packet loss = 100%". Users normally directed to eqsin, mostly in the APAC region, could only read Wikipedia pages already cached in eqsin.

   Network unreachable

   Trigger

   Planned maintenance by the transport provider took down eqsin's second transport link while the first was still down from a long fiber cut.

   Root cause

   Dependency failure

   Component

   Backbone network (WAN)BGP routingCDN and edge network

   Contributing factor

   Single point of failureIgnored early signal

   Detection

   Automated monitoring: Icinga BGP status alerts for cr3-eqsin at 16:37 and cr2-eqsin at 16:44, followed by a flood of host-down alerts and three pages at the same time.

   Remediation

   • Depooled eqsin from DNS
   • Repooled eqsin

   Action items

   • Create a backup GRE tunnel for eqsin (T327265)
   • Ensure the long-down transport link comes back up properly once it is fixed (T322529)
   • Automatically parse maintenance notifications and alert on conflicting maintenance (T230835)

   Lessons

   The site was depooled quickly, and "depool first, investigate later" was the right attitude. A flood of host-down alerts usually means a network problem. eqsin stayed too long on a single working transport link, SREs did not catch that the downtime of the two transport links overlapped, and the provider's planned maintenance was not recorded in a task or reminder.

   View case
3. IL-0358 Packet Public Cloud locations had high latency and unreachable paths from routing loops and blackholed traffic

   Packet2019

   Symptoms

   At about 08:49 UTC on 2 June 2019, Packet's internal network monitoring alerted on serious reachability and performance problems in several of Packet's global datacenter locations. The first investigation found routing loops and traffic being blackholed to specific internal and Internet destinations.

   Network unreachablePacket lossHigh latency

   Trigger

   A trans-continental US backbone link in the network of Packet's upstream provider Zayo had a hardware failure and blackholed IP traffic routed over it.

   Root cause

   Dependency failure

   Component

   Backbone network (WAN)BGP routingNetwork control plane (SDN)

   Contributing factor

   –

   Detection

   Packet's internal network monitoring alerted at 08:49 UTC on reachability issues in several global datacenter locations, and engineers confirmed them at 08:52.

   Remediation

   • Migrated traffic to an alternative network and removed Zayo from the provider mix
   • Applied route filters to return service to normal
   • Optimised specific routing policies to improve latency across the backbone with the new traffic mix

   Action items

   • Expand internal connectivity monitoring and alerting of on-call operations teams
   • Expand network automation and routing so traffic can be balanced across all available backbone paths and an operator can trigger automated re-routes across all Packet datacenters in a major provider incident
   • Make Packet's provisioning systems less dependent on third party network providers and inter-datacenter connectivity

   View case
4. IL-0258 FastMail was unreachable for almost two hours when the GRE tunnels carrying its traffic from Level 3 went down

   Fastmail2016

   Symptoms

   At 4.50am Melbourne time on Sunday 7 February (evening of 6 February UTC), FastMail became unavailable and stayed unreachable for almost two hours. Traffic to FastMail's public IP range never made it into FastMail's own network.

   Service outageNetwork unreachable

   Trigger

   The GRE tunnels between Level 3 and NYI, which carried FastMail's filtered traffic into its network, went down.

   Root cause

   Dependency failure

   Component

   DDoS protection serviceBGP routing

   Contributing factor

   Limited debug access

   Detection

   NYI, the datacentre provider, noticed the failure quickly.

   Remediation

   • NYI escalated with Level 3's support until FastMail's public IP ranges were pointed back to NYI

   Action items

   • Go back to DDoS protection from Black Lotus under NYI's explicit control
   • Build nginx frontends in the Los Angeles datacentre that reach NYI over the secondary network, and point DNS to them if NYI's primary network fails
   • Add hardware in Amsterdam and change the delivery system so it depends less on NYI
   • Work with NYI to understand its processes for detecting and responding to network faults

   Lessons

   Much of FastMail's and NYI's reliability had come from NYI controlling its own infrastructure. With protection always on at Level 3, NYI could not restore the routes itself, and FastMail calls the resulting delay in restoring service unacceptable.

   View case

By root cause1