Cases1 case

1 to 1 of 1

1. IL-0259 Outbound packets from Fastmail did not reach part of the internet for about 19 hours, cutting off 3-5% of customers

   Fastmail2023

   Symptoms

   From about 03:00 UTC on 30 June 2023, Fastmail and a fraction of the internet could not communicate with each other, and Fastmail first heard of it from customers through support tickets and social media. The problem lasted about 19 hours and affected an estimated 3-5% of customers as well as some email servers.

   Network unreachable

   Root cause

   Dependency failure

   Component

   ISP and transit networks

   Contributing factor

   Single point of failureMonitoring blind spot

   Detection

   Customers told Fastmail through support tickets and social media. Fastmail's own monitors in multiple locations and its staff were not on affected networks and saw nothing.

   Action items

   • Select a provider for a second transit connection directly into Fastmail's servers, via Megaport or a provider present in 365's New Jersey datacenter, so that Fastmail controls the outbound path and can route around a network with problems

   Lessons

   With a single path for outbound traffic, Fastmail could not route around a provider's network problem and could not fix it itself. Being able to choose between several outbound paths would let Fastmail get everyone back online faster.

   View case

By root cause1